# Collapsed Gibbs sampler for Generalized Relational Topic Models with Data Augmentation

  $
      \DeclareMathOperator{\dir}{Dirichlet}
      \DeclareMathOperator{\dis}{Discrete}
      \DeclareMathOperator{\normal}{Normal}
      \DeclareMathOperator{\ber}{Bernoulli}
      \DeclareMathOperator{\diag}{diag}
      \DeclareMathOperator{\Betaf}{B}
      \DeclareMathOperator{\Gammaf}{\Gamma}
      \DeclareMathOperator{\PG}{PG}
      \DeclareMathOperator{\v}{vec}
      \newcommand{\norm}[1]{\left\| #1 \right\|}
      \newcommand{\cp}[2]{p \left( #1 \middle| #2 \right)}
      \newcommand{\cN}[2]{\mathscr{N} \left( #1 \middle| #2 \right)}
      \newcommand{\cpsi}[2]{\psi \left( #1 \middle| #2 \right)}
      \newcommand{\cPsi}[2]{\Psi \left( #1 \middle| #2 \right)}
      \newcommand{\etd}[1]{\mathbf{z}^{(#1)}}
      \newcommand{\etdT}[1]{\left. \mathbf{z}^{(#1)} \right.^T}
      \newcommand{\Etd}[2]{\mathbf{z}^{(#1, #2)}}
      \newcommand{\sumetd}{\mathbf{z}}
      \newcommand{\one}{\mathbf{1}}
      \newcommand{\Eta}{H}
      \newcommand{\eHe}{\etdT{d} \Eta \etd{d'}}
  $

Here is the collapsed Gibbs sampler for Chen et al.'s [generalized relational topic models with data augmentation](http://ijcai.org/papers13/Papers/IJCAI13-192.pdf).  I am building on the [collapsed Gibbs sampler](http://nbviewer.savvysherpa.com/github/gp-0058-clt-at-toys-r-us/relational-topic-models/blob/master/blslda.ipynb) I wrote for binary logistic supervised latent Dirichlet allocation.

The generative model for RTMs is as follows:

$$\begin{align}
\theta^{(d)} &\sim \dir(\alpha) &\text{(topic distribution for document $d \in \{1, \ldots, D\}$)} \\
\phi^{(k)} &\sim \dir(\beta) &\text{(term distribution for topic $k \in \{1, \ldots, K\}$)} \\
z_n^{(d)} \mid \theta^{(d)} &\sim \dis \left( \theta^{(d)} \right) &\text{(topic of $n$th token of document $d$, $n \in \{1, \ldots, N^{(d)}\}$)} \\
w_n^{(d)} \mid \phi^{(z_n^{(d)})} &\sim \dis \left( \phi^{(z_n^{(d)})} \right) &\text{(term of $n$th token of document $d$, $n \in \{1, \ldots, N^{(d)}\}$)} \\
\Eta_{k, k'} &\sim \normal \left( \mu, \nu^2 \right) &\text{(regression coefficients for topic pairs $k, k' \in \{1, \ldots, K\}$)} \\
y^{(d, d')} \mid \Eta, \etd{d}, \etd{d'} &\sim \ber \left(
\frac{ \exp \left( \eHe \right) }{ 1 + \exp \left( \eHe \right) } \right)
&\text{(link indicator for documents $d, d' \in \{1, \ldots, D\}$)}
\end{align}$$

where each token can be any one of $V$ terms in our vocabulary, $\etd{d}$ is the empirical topic distribution of document $d$, and $\circ$ is the Hadamard (element-wise) product.

Plate notation for relational topic models.

This diagram should replace $\beta_k$ with $\phi^{(k)}$, and each $\phi^{(k)}$ should be dependent on a single $\beta$.

Following [Chen et al. 2013](http://ijcai.org/papers13/Papers/IJCAI13-192.pdf), the regularized pseudo-likelihood for the link variable $y^{(d, d')}$, with regularization parameter $b \ge 0$, can be written

$$\begin{align}
\cpsi{y^{(d, d')}}{\Eta, \etd{d}, \etd{d'}, b}
&= \cp{y^{(d, d')}}{\Eta, \etd{d}, \etd{d'}}^b
\\ &= \left( \frac{\exp \left( \eHe \right)^{y^{(d, d')}}}{ 1 + \exp \left( \eHe \right)} \right)^b
\\ &= \frac{\exp \left( b y^{(d, d')} \eHe \right)}
{ \left( \exp \left( -\frac{\eHe}{2} \right) + \exp \left( \frac{\eHe}{2} \right) \right)^b \exp \left( \frac{b}{2} \eHe \right) }
\\ &= 2^{-b} \exp \left( b \left( y^{(d, d')} - \frac{1}{2} \right) \left( \eHe \right) \right) \cosh \left( \frac{ \eHe }{2} \right)^{-b}
\\ &= 2^{-b} \exp \left( b \left( y^{(d, d')} - \frac{1}{2} \right) \left( \eHe \right) \right)
\int_0^\infty \exp \left( -\frac{ \left( \eHe \right)^2 }{2} \omega^{(d, d')} \right)
\cp{\omega^{(d, d')}}{b, 0} d\omega^{(d, d')}
\end{align}$$

where $\omega^{(d, d')}$ is a Polya-Gamma distributed variable with parameters $b = b$ and $c = 0$ (see [Polson et al. 2012](http://arxiv.org/pdf/1205.0310v3.pdf) for details).  This means that, for each pair of documents $d$ and $d'$, the pseudo-likelihood of $y^{(d, d')}$ is actually a mixture of Gaussians with respect to the Polya-Gamma distribution $\PG(b, 0)$.  Therefore, the joint pseudo-likelihood of $y^{(d, d')}$ and $\omega^{(d, d')}$ can be written

$$\cPsi{y^{(d, d')}, \omega^{(d, d')}}{\Eta, \etd{d}, \etd{d'}, b}
= 2^{-b} \exp \left( \kappa^{(d, d')} \zeta^{(d, d')} - \frac{ \omega^{(d, d')} }{2} (\zeta^{(d, d')})^2 \right) \cp{\omega^{(d, d')}}{b, 0}.$$

where $\kappa^{(d, d')} = b(y^{(d, d')} - 1/2)$ and $\zeta^{(d, d')} = \eHe$.  The joint probability distribution can therefore be factored as follows:

$$\begin{align}
\cp{\theta, \phi, z, w, \Eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b}
&=
\prod_{k=1}^{K} \cp{\phi^{(k)}}{\beta}
\prod_{d=1}^{D} \cp{\theta^{(d)}}{\alpha}
\prod_{n=1}^{N^{(d)}} \cp{z_n^{(d)}}{\theta^{(d)}} \cp{w_n^{(d)}}{\phi^{(z_n^{(d)})}}
\\ & \quad \times \prod_{k_1=1}^{K} \prod_{k_2=1}^{K} \cp{\Eta_{k_1, k_2}}{\mu, \nu^2}
\prod_{d_1=1}^D \prod_{\substack{d_2=1 \\ d_2 \neq d_1}}^D \cPsi{y^{(d_1, d_2)}, \omega^{(d_1, d_2)}}{\Eta, \etd{d_1}, \etd{d_2}, b}
\\ &=
\prod_{k=1}^{K} \frac{\Betaf(b^{(k)} + \beta)}{\Betaf(\beta)} \cp{\phi^{(k)}}{b^{(k)} + \beta}
\prod_{d=1}^{D} \frac{\Betaf(a^{(d)} + \alpha)}{\Betaf(\alpha)} \cp{\theta^{(d)}}{a^{(d)} + \alpha}
\\ &\quad \times
\prod_{k_1=1}^{K} \prod_{k_2=1}^{K} \cN{\Eta_{k_1, k_2}}{\mu, \nu^2}
\prod_{d_1=1}^D \prod_{\substack{d_2=1 \\ d_2 \neq d_1}}^D 2^{-b} \exp \left( \kappa^{(d_1, d_2)} \zeta^{(d_1, d_2)} - \frac{ \omega^{(d_1, d_2)} }{2} (\zeta^{(d_1, d_2)})^2 \right) \cp{\omega^{(d_1, d_2)}}{b, 0}
\end{align}$$

where $a_k^{(d)}$ is the number of tokens in document $d$ assigned to topic $k$, $b_v^{(k)}$ is the number of tokens equal to term $v$ and assigned to topic $k$, and $\Betaf$ is the [multivariate Beta function](https://en.wikipedia.org/wiki/Beta_function#Multivariate_beta_function). Marginalizing out $\theta$ and $\phi$ by integrating with respect to each $\theta^{(d)}$ and $\phi^{(k)}$ over their respective sample spaces yields

$$\begin{align}
\cp{z, w, \Eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b} &=
\prod_{k=1}^{K} \frac{\Betaf(b^{(k)} + \beta)}{\Betaf(\beta)}
\prod_{d=1}^{D} \frac{\Betaf(a^{(d)} + \alpha)}{\Betaf(\alpha)}
\\ &\quad\quad \times \prod_{k_1=1}^{K} \prod_{k_2=1}^{K} \cN{\Eta_{k_1, k_2}}{\mu, \nu^2}
\prod_{d_1=1}^D \prod_{\substack{d_2=1 \\ d_2 \neq d_1}}^D 2^{-b} \exp \left( \kappa^{(d_1, d_2)} \zeta^{(d_1, d_2)} - \frac{ \omega^{(d_1, d_2)} }{2} (\zeta^{(d_1, d_2)})^2 \right) \cp{\omega^{(d_1, d_2)}}{b, 0}
\\ &=
\cp{w}{z, \beta} \cp{z}{\alpha} \cp{\Eta}{\mu, \nu^2} \cPsi{y, \omega}{\Eta, z, b}.
\end{align}$$

See my [LDA notebook](http://nbviewer.savvysherpa.com/github/bearnshaw/ml-demos/blob/master/lda_gibbs_sampling_cython.ipynb) for step-by-step details of the previous two calculations.

Our goal is to calculate the posterior distribution

$$\cp{z, \Eta, \omega}{w, y, \alpha, \beta, \mu, \nu^2, b} =
\frac{\cp{z, w, \Eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b}}
{\sum_{z'} \iint \cp{z', w, \Eta', y, \omega{'}}{\alpha, \beta, \mu, \nu^2, b} d\Eta' d\omega{'}}$$

in order to infer the topic assignments $z$ and regression coefficients $\Eta$ from the given term assignments $w$ and link data $y$.  Since calculating this directly is infeasible, we resort to collapsed Gibbs sampling.  The sampler is "collapsed" because we marginalized out $\theta$ and $\phi$, and will estimate them from the topic assignments $z$:

$$\hat\theta_k^{(d)} = \frac{a_k^{(d)} + \alpha_k}{\sum_{k'=1}^K \left(a_{k'}^{(d)} + \alpha_{k'} \right)},\quad
\hat\phi_v^{(k)} = \frac{b_v^{(k)} + \beta_v}{\sum_{v'=1}^V \left(b_{v'}^{(k)} + \beta_{v'} \right)}.$$

Gibbs sampling requires us to compute the full conditionals for each $z_n^{(d)}$, $\omega^{(d, d')}$ and $\Eta_{k, k'}$.  For example, we need to calculate, for all $n$, $d$ and $k$,

$$\begin{align}
\cp{z_n^{(d)} = k}{z \setminus z_n^{(d)}, w, H, y, \omega, \alpha, \beta, \mu, \nu^2, b}
&\propto
\cp{z_n^{(d)} = k, z \setminus z_n^{(d)}, w, H, y, \omega}{\alpha, \beta, \mu, \nu^2, b}
\\ &\propto
\frac{b_{w_n^{(d)}}^{(k)} \setminus z_n^{(d)} + \beta_{w_n^{(d)}}}{ \sum_{v=1}^V \left( b_v^{(k)} \setminus z_n^{(d)} + \beta_v\right)}
\left( a_k^{(d)} \setminus z_n^{(d)} + \alpha_k \right)
\prod_{d_1=1}^{D} \prod_{\substack{d_2=1 \\ d_2 \neq d_1}}^{D} \exp \left( \kappa^{(d_1, d_2)} \zeta^{(d_1, d_2)} - \frac{ \omega^{(d_1, d_2)} }{2} (\zeta^{(d_1, d_2)})^2 \right)
\\ &\propto
\frac{b_{w_n^{(d)}}^{(k)} \setminus z_n^{(d)} + \beta_{w_n^{(d)}}}{ \sum_{v=1}^V \left( b_v^{(k)} \setminus z_n^{(d)} + \beta_v\right)}
\left( a_k^{(d)} \setminus z_n^{(d)} + \alpha_k \right)
\\ &\quad\quad\times
\exp \left( \sum_{\substack{d_1=1 \\ d_1 \neq d}}^{D} \left[ \left( \kappa^{(d_1, d)} - \omega^{(d_1, d)} ( \zeta^{(d_1, d)} \setminus z_n^{(d)}) \right) \frac{H_{:, k}^T \etd{d_1}}{N^{(d)}}
- \frac{ \omega^{(d_1, d)} }{2} \left( \frac{H_{:, k}^T \etd{d_1}}{N^{(d)}} \right)^2 \right] \right.
\\ &\quad\quad\quad\quad +
\left. \sum_{\substack{d_2=1 \\ d_2 \neq d}}^{D} \left[ \left( \kappa^{(d, d_2)} - \omega^{(d, d_2)} ( \zeta^{(d, d_2)} \setminus z_n^{(d)}) \right) \frac{H_{k, :} \etd{d_2}}{N^{(d)}}
- \frac{ \omega^{(d, d_2)} }{2} \left( \frac{H_{k, :} \etd{d_2}}{N^{(d)}} \right)^2 \right] \right)
\end{align}$$

where the "set-minus" notation $\cdot \setminus z_n^{(d)}$ denotes the variable the notation is applied to with the entry $z_n^{(d)}$ removed (again, see my [LDA notebook](http://nbviewer.savvysherpa.com/github/bearnshaw/ml-demos/blob/master/lda_gibbs_sampling_cython.ipynb) for details).  This final proportionality is true since

$$\begin{align}
\prod_{d_1=1}^{D} \prod_{\substack{d_2=1 \\ d_2 \neq d_1}}^{D} \exp \left( \kappa^{(d_1, d_2)} \zeta^{(d_1, d_2)} - \frac{ \omega^{(d_1, d_2)} }{2} (\zeta^{(d_1, d_2)})^2 \right)
&=
\prod_{d_1=1}^{D} \prod_{\substack{d_2=1 \\ d_2 \neq d_1}}^{D} \exp \left( \kappa^{(d_1, d_2)} \left( \zeta^{(d_1, d_2)} \setminus z_n^{(d)} + \Delta_{d, d_1, d_2}^{(k)} \right)
- \frac{ \omega^{(d_1, d_2)} }{2} \left( \zeta^{(d_1, d_2)} \setminus z_n^{(d)} + \Delta_{d, d_1, d_2}^{(k)} \right)^2 \right)
\\ &\propto
\prod_{\substack{d_1=1 \\ d_1 \neq d}}^{D} \exp \left( \kappa^{(d_1, d)} \left( \zeta^{(d_1, d)} \setminus z_n^{(d)} + \Delta_{d, d_1, d}^{(k)} \right)
- \frac{ \omega^{(d_1, d)} }{2} \left( \zeta^{(d_1, d)} \setminus z_n^{(d)} + \Delta_{d, d_1, d}^{(k)} \right)^2 \right)
\\ &\quad\quad\times
\prod_{\substack{d_2=1 \\ d_2 \neq d}}^{D} \exp \left( \kappa^{(d, d_2)} \left( \zeta^{(d, d_2)} \setminus z_n^{(d)} + \Delta_{d, d, d_2}^{(k)} \right)
- \frac{ \omega^{(d, d_2)} }{2} \left( \zeta^{(d, d_2)} \setminus z_n^{(d)} + \Delta_{d, d, d_2}^{(k)} \right)^2 \right)
\\ &\propto
\exp \left( \sum_{\substack{d_1=1 \\ d_1 \neq d}}^{D} \left[ \left( \kappa^{(d_1, d)} - \omega^{(d_1, d)} ( \zeta^{(d_1, d)} \setminus z_n^{(d)}) \right) \frac{H_{:, k}^T (\etd{d_1} \setminus z_n^{(d)})}{N^{(d)}}
- \frac{ \omega^{(d_1, d)} }{2} \left( \frac{H_{:, k}^T (\etd{d_1} \setminus z_n^{(d)})}{N^{(d)}} \right)^2 \right] \right.
\\ &\quad\quad +
\left. \sum_{\substack{d_2=1 \\ d_2 \neq d}}^{D} \left[ \left( \kappa^{(d, d_2)} - \omega^{(d, d_2)} ( \zeta^{(d, d_2)} \setminus z_n^{(d)}) \right) \frac{H_{k, :} (\etd{d_2} \setminus z_n^{(d)})}{N^{(d)}}
- \frac{ \omega^{(d, d_2)} }{2} \left( \frac{H_{k, :} (\etd{d_2} \setminus z_n^{(d)})}{N^{(d)}} \right)^2 \right] \right)
\\ &=
\exp \left( \sum_{\substack{d_1=1 \\ d_1 \neq d}}^{D} \left[ \left( \kappa^{(d_1, d)} - \omega^{(d_1, d)} ( \zeta^{(d_1, d)} \setminus z_n^{(d)}) \right) \frac{H_{:, k}^T \etd{d_1}}{N^{(d)}}
- \frac{ \omega^{(d_1, d)} }{2} \left( \frac{H_{:, k}^T \etd{d_1}}{N^{(d)}} \right)^2 \right] \right.
\\ &\quad\quad +
\left. \sum_{\substack{d_2=1 \\ d_2 \neq d}}^{D} \left[ \left( \kappa^{(d, d_2)} - \omega^{(d, d_2)} ( \zeta^{(d, d_2)} \setminus z_n^{(d)}) \right) \frac{H_{k, :} \etd{d_2}}{N^{(d)}}
- \frac{ \omega^{(d, d_2)} }{2} \left( \frac{H_{k, :} \etd{d_2}}{N^{(d)}} \right)^2 \right] \right)
\end{align}$$

where

$$\Delta_{d, d_1, d_2}^{(k)} = \delta_{d, d_1} \frac{H_{k, :} (\etd{d_2} \setminus z_n^{(d)})}{N^{(d)}} + \delta_{d, d_2} \frac{H_{:, k}^T (\etd{d_1} \setminus z_n^{(d)})}{N^{(d)}},$$

$\delta_{d, d'}$ is the [Kronecker delta](https://en.wikipedia.org/wiki/Kronecker_delta), and $H_{k, :}$ and $H_{:, k}$ are the $k$th row and column of $H$, respectively.  The first proportionality is a result of the fact that $\Delta_{d, d_1, d_2}^{(k)}$ is nonzero only when $d = d_1$ or $d = d_2$.  The last equality follows from the fact that $d \neq d_1$ in the first summation and $d \neq d_2$ in the second.

In order to calculate the full conditional for $H$, let $\eta = (H_{:,1}^T \cdots H_{:, K}^T)^T$ be the vector of concatenated columns of $H$, $Z = (\etd{1, 1} \cdots \etd{D, D})$ be the matrix whose columns are the vectors $\etd{d, d'} = \etd{d'} \otimes \etd{d}$, where $\otimes$ is the [Kronecker product](https://en.wikipedia.org/wiki/Kronecker_product), $\Omega = \diag(\omega^{(1,1)}, \ldots, \omega^{(D,D)})$ be the diagonal matrix whose diagonal entries are $\omega^{(d, d')}$, $I$ be the identity matrix, and $\one$ be the vector of ones, and note that

$$\prod_{k_1=1}^{K} \prod_{k_2=1}^{K} \cN{H_{k_1, k_2}}{\mu, \nu^2} = \cN{\eta}{\mu \one, \nu^2 I}$$
$$\prod_{d_1=1}^{D} \prod_{\substack{d_2=1 \\ d_2 \neq d_1}}^{D} \exp \left( \kappa^{(d_1, d_2)} \zeta^{(d_1, d_2)} - \frac{ \omega^{(d_1, d_2)} }{2} (\zeta^{(d_1, d_2)})^2 \right)
= \exp \left( \eta^T Z \kappa - \frac{1}{2} \eta^T Z \Omega Z^T \eta \right)
\propto \cN{\eta}{(Z \Omega Z^T)^{-1} Z \kappa, (Z \Omega Z^T)^{-1}}.$$

Therefore

$$\begin{align}
\cp{\eta}{z, w, y, \omega, \alpha, \beta, \mu, \nu^2, b}
&\propto
\cp{z, w, \eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b}
\\ &\propto
\cN{\eta}{\mu \one, \nu^2 I} \cN{\eta}{(Z \Omega Z^T)^{-1} Z \kappa, (Z \Omega Z^T)^{-1}}
\\ &\propto
\cN{\eta}{\Sigma \left( \frac{\mu}{\nu^2} \one + Z \kappa \right), \Sigma}
\end{align}$$

where $\Sigma^{-1} = \nu^{-2} I + Z \Omega Z^T$ (see Section 8.1.8 of the [Matrix Cookbook](http://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf)).

We also need to calculate the full conditional for $\omega$.  We calculate

$$\begin{align}
\cp{\omega}{z, w, H, y, \alpha, \beta, \mu, \nu^2, b}
&\propto
\cp{z, w, H, y, \omega}{\alpha, \beta, \mu, \nu^2, b}
\\ &\propto
\prod_{d_1=1}^{D} \prod_{\substack{d_2=1 \\ d_2 \neq d_1}}^{D} \exp \left( - \frac{ \omega^{(d_1, d_2)} }{2} (\zeta^{(d_1, d_2)})^2 \right) \cp{\omega^{(d_1, d_2)}}{b, 0}
\\ &=
\prod_{d_1=1}^{D} \prod_{\substack{d_2=1 \\ d_2 \neq d_1}}^{D} \cp{\omega^{(d_1, d_2)}}{b, \zeta^{(d_1, d_2)}}
\end{align}$$

that is, $\omega^{(d_1, d_2)} \sim \PG(b, \eHe)$ for each pair of documents $d_1$ and $d_2$.  We sample from the Polya-Gamma distribution according to the method of [Polson et al. 2012](http://arxiv.org/pdf/1205.0310.pdf), implemented for Python 3 in this [code repo](https://github.com/Savvysherpa/pypolyagamma).

## Graphical test

```python
%matplotlib inline

from modules.helpers import plot_images
from functools import partial
from sklearn.metrics import (roc_auc_score, roc_curve)
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

imshow = partial(plt.imshow, cmap='gray', interpolation='nearest', aspect='auto')
sns.set(style='white')
```

### Generate topics

We assume a vocabulary of 25 terms, and create ten "topics", where each topic assigns exactly 5 consecutive terms equal probability.

```python
V = 25
K = 10
N = 100
D = 1000
```

```python
topics = []
topic_base = np.concatenate((np.ones((1, 5)) * 0.2, np.zeros((4, 5))), axis=0).ravel()
for i in range(5):
    topics.append(np.roll(topic_base, i * 5))
topic_base = np.concatenate((np.ones((5, 1)) * 0.2, np.zeros((5, 4))), axis=1).ravel()
for i in range(5):
    topics.append(np.roll(topic_base, i))
topics = np.array(topics)
plt.figure(figsize=(10, 5))
plot_images(plt, topics, (5, 5), layout=(2, 5), figsize=(10, 5))
```

### Generate documents from topics

We generate 1,000 documents from these 10 topics by sampling 1,000 topic distributions, one for each document, from a Dirichlet distribution with parameter $\alpha = (1, \ldots, 1)$.

```python
alpha = np.ones(K)
np.random.seed(42)
thetas = np.random.dirichlet(alpha, size=D)
topic_assignments = np.array([np.random.choice(range(K), size=100, p=theta)
                              for theta in thetas])
word_assignments = np.array([[np.random.choice(range(V), size=1, p=topics[topic_assignments[d, n]])[0]
                              for n in range(N)] for d in range(D)])
doc_term_matrix = np.array([np.histogram(word_assignments[d], bins=V, range=(0, V - 1))[0] for d in range(D)])
imshow(doc_term_matrix)
```

### Generate document network

Create a document network from the documents by applying $\psi$ and applying a threshold $\psi_0$.

```python
from itertools import product
from sklearn.cross_validation import StratifiedKFold
```

```python
# choose parameter values
mu = 0.
nu2 = 1.
np.random.seed(14)
H = np.random.normal(loc=mu, scale=nu2, size=(K, K))
zeta = pd.DataFrame([(i, j, np.dot(np.dot(thetas[i], H), thetas[j])) for i, j in product(range(D), repeat=2)],
                    columns=('tail', 'head', 'zeta'))
_ = zeta.zeta.hist(bins=50)
```

```python
# choose parameter values
zeta['y'] = (zeta.zeta >= 0).astype(int)
# plot histogram of responses
print('positive examples {} ({:.1f}%)'.format(zeta.y.sum(), zeta.y.sum() / D / D * 100))
_ = zeta.y.hist()
```

```python
y = zeta[['tail', 'head', 'y']].values
```

```python
skf = StratifiedKFold(y[:, 2], n_folds=100)
_, train_idx = next(iter(skf))
train_idx.shape
```

    (10001,)

### Estimate parameters

```python
from slda.topic_models import GRTM
```

```python
_K = 10
_alpha = alpha[:_K]
_beta = np.repeat(0.01, V)
_mu = mu
_nu2 = nu2
_b = 1.
n_iter = 500
grtm = GRTM(_K, _alpha, _beta, _mu, _nu2, _b, n_iter, seed=42)
```

```python
%%time
grtm.fit(doc_term_matrix, y[train_idx])
```

    CPU times: user 9min 9s, sys: 917 ms, total: 9min 10s
    Wall time: 9min 11s

```python
plot_images(plt, grtm.phi, (5, 5), (2, 5), figsize=(10, 5))
```

```python
topic_order = [4, 7, 3, 1, 0, 9, 5, 2, 8]
plot_images(plt, grtm.phi[topic_order], (5, 5), (2, 5), figsize=(10, 5))
```

```python
burnin = -1
mean_final_lL = grtm.loglikelihoods[burnin:].mean()
print(mean_final_lL)
plt.plot(grtm.loglikelihoods, label='mean final LL {:.2f}'.format(mean_final_lL))
_ = plt.legend()
```

```python
imshow(grtm.theta)
```

```python
H_pred = grtm.H[burnin:].mean(axis=0)
_ = plt.hist(H_pred.ravel(), bins=20)
```

```python
_ = plt.hist(H.ravel(), bins=20)
```

### Predict edges on pairs of test documents

Create 1,000 test documents using the same generative process as our training documents.

```python
np.random.seed(42^2)
thetas_test = np.random.dirichlet(alpha, size=D)
topic_assignments_test = np.array([np.random.choice(range(K), size=100, p=theta)
                                   for theta in thetas_test])
word_assignments_test = np.array([[np.random.choice(range(V), size=1, p=topics[topic_assignments_test[d, n]])[0]
                                   for n in range(N)] for d in range(D)])
doc_term_matrix_test = np.array([np.histogram(word_assignments_test[d], bins=V, range=(0, V - 1))[0] for d in range(D)])
imshow(doc_term_matrix_test)
```

Learn their topic distributions using the model trained on the training documents, then calculate the actual and predicted values of $\psi$.  For predicted $\psi$, estimate $\eta$ as the mean of our samples of $\eta$ after burn-in.

```python
def bern_param(theta1, theta2, H):
    zeta = np.dot(np.dot(theta1, H), theta2)
    return np.exp(zeta) / (1 + np.exp(zeta))
```

```python
thetas_test_grtm = grtm.transform(doc_term_matrix_test)
p_test = np.zeros(D * D)
p_test_grtm = np.zeros(D * D)
for n, i in enumerate(product(range(D), range(D))):
    p_test[n] = bern_param(thetas_test[i[0]], thetas_test[i[1]], H)
    p_test_grtm[n] = bern_param(thetas_test_grtm[i[0]], thetas_test_grtm[i[1]], H_pred)
```

Measure the goodness of our prediction by the [area under the associated ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve).

```python
y_test = (p_test > 0.5).astype(int)
y_grtm = p_test_grtm
fpr, tpr, _ = roc_curve(y_test, y_grtm)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_grtm))))
_ = plt.legend(loc='best')
```

### Learn topics, then learn classifier

```python
from slda.topic_models import LDA
```

```python
lda = LDA(_K, _alpha, _beta, n_iter, seed=42)
```

```python
%%time
lda.fit(doc_term_matrix)
```

    CPU times: user 2.24 s, sys: 9.37 ms, total: 2.24 s
    Wall time: 2.24 s

```python
plot_images(plt, lda.phi, (5, 5), (1, 5), figsize=(10, 5))
```

```python
plt.plot(lda.loglikelihoods)
```

```python
imshow(lda.theta)
```

```python
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
```

Compute Hadamard products between learned topic distributions for training and test documents.

```python
%%time
thetas_test_lda = lda.transform(doc_term_matrix_test)
```

    CPU times: user 112 ms, sys: 1.34 ms, total: 114 ms
    Wall time: 113 ms

```python
lda_train = np.zeros((D * D, _K * _K))
lda_test = np.zeros((D * D, _K * _K))
for n, i in enumerate(product(range(D), range(D))):
    lda_train[n] = np.kron(lda.theta[i[0]], lda.theta[i[1]])
    lda_test[n] = np.kron(thetas_test_lda[i[0]], thetas_test_lda[i[1]])
```

#### Logistic regression

- Train logistic regression on training data
- calculate probability of an edge for each pair of test documents, and
- measure the goodness of our prediction by computing the [area under the ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve).

```python
_C_grid = np.arange(1, 202, 10)
roc_auc_scores = []
for _C in _C_grid:
    print('Training Logistic Regression with C = {}'.format(_C))
    _lr = LogisticRegression(fit_intercept=False, C=_C)
    _lr.fit(lda_train, zeta.y)
    _y_lr = _lr.predict_proba(lda_test)[:, 1]
    roc_auc_scores.append(roc_auc_score(y_test, _y_lr))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_C_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_C_grid, roc_auc_scores)
```

```python
lr = LogisticRegression(fit_intercept=False, C=11)
lr.fit(lda_train, zeta.y)
y_lr = lr.predict_proba(lda_test)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_lr)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_lr))))
_ = plt.legend(loc='best')
```

#### Gradient boosted trees

- Train gradient boosted trees on training data
- calculate probability of an edge for each pair of test documents, and
- measure the goodness of our prediction by computing the [area under the ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve).

```python
_C_grid = np.arange(1, 4)
roc_auc_scores = []
for _C in _C_grid:
    print('Training Logistic Regression with C = {}'.format(_C))
    _gbc = GradientBoostingClassifier(max_depth=_C)
    _gbc.fit(lda_train, zeta.y)
    _y_gbc = _gbc.predict_proba(lda_test)[:, 1]
    roc_auc_scores.append(roc_auc_score(y_test, _y_gbc))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_C_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_C_grid, roc_auc_scores)
```

```python
gbc = GradientBoostingClassifier(max_depth=3)
gbc.fit(lda_train, zeta.y)
y_gbc = gbc.predict_proba(lda_test)[:, 1]
fpr_gbc, tpr_gbc, _ = roc_curve(y_test, y_gbc)
plt.plot(fpr_gbc, tpr_gbc, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_gbc))))
plt.legend(loc='best')
```

### Use GRTM topics

```python
grtm_train = np.zeros((D * D, _K * _K))
grtm_test = np.zeros((D * D, _K * _K))
for n, i in enumerate(product(range(D), range(D))):
    grtm_train[n] = np.kron(grtm.theta[i[0]], grtm.theta[i[1]])
    grtm_test[n] = np.kron(thetas_test_grtm[i[0]], thetas_test_grtm[i[1]])
```

```python
_C_grid = np.arange(1, 52, 10)
roc_auc_scores = []
for _C in _C_grid:
    print('Training Logistic Regression with C = {}'.format(_C))
    _lr = LogisticRegression(fit_intercept=False, C=_C)
    _lr.fit(grtm_train, zeta.y)
    _y_lr = _lr.predict_proba(grtm_test)[:, 1]
    roc_auc_scores.append(roc_auc_score(y_test, _y_lr))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_C_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_C_grid, roc_auc_scores)
```

```python
lr = LogisticRegression(fit_intercept=False, C=51)
lr.fit(grtm_train, zeta.y)
y_lr = lr.predict_proba(grtm_test)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_lr)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_lr))))
_ = plt.legend(loc='best')
```

## Conclusion

The relational topic model performs better than either logistic regression or gradient boosted trees trained on the LDA topics, but worse than a logistic regression trained on the GRTM topics, which just shows how much better the GRTM topics are!

```python

```


# Optimización media-varianza

La **teoría de portafolios** es una de los avances más importantes en las finanzas modernas e inversiones.
- Apareció por primera vez en un [artículo corto](https://www.math.ust.hk/~maykwok/courses/ma362/07F/markowitz_JF.pdf) llamado "Portfolio Selection" en la edición de Marzo de 1952 de "the Journal of Finance".
- Escrito por un desconocido estudiante de la Universidad de Chicago, llamado Harry Markowitz.
- Escrito corto (sólo 14 páginas), poco texto, fácil de entender, muchas gráficas y unas cuantas referencias.
- No se le prestó mucha atención hasta los 60s.

Finalmente, este trabajo se convirtió en una de las más grandes ideas en finanzas, y le dió a Markowitz el Premio Nobel casi 40 años después.
- Markowitz estaba incidentalmente interesado en los mercados de acciones e inversiones.
- Estaba más bien interesado en entender cómo las personas tomaban sus mejores decisiones cuando se enfrentaban con "trade-offs".
- Principio de conservación de la miseria. O, dirían los instructores de gimnasio: "no pain, no gain".
- Si queremos más de algo, tenemos que perder en algún otro lado.
- El estudio de este fenómeno era el que le atraía a Markowitz.

De manera que nadie se hace rico poniendo todo su dinero en la cuenta de ahorros. La única manera de esperar altos rendimientos es si se toma bastante riesgo. Sin embargo, riesgo significa también la posibilidad de perder, tanto como ganar.

Pero, ¿qué tanto riesgo es necesario?, y ¿hay alguna manera de minimizar el riesgo mientras se maximizan las ganancias?
- Markowitz básicamente cambió la manera en que los inversionistas pensamos acerca de esas preguntas.
- Alteró completamente la práctica de la administración de inversiones.
- Incluso el título de su artículo era innovador. Portafolio: una colección de activos en lugar de tener activos individuales.
- En ese tiempo, un portafolio se refería a una carpeta de piel.
- En el resto de este módulo, nos ocuparemos de la parte analítica de la teoría de portafolios, la cual puede ser resumida en dos frases:
 - No pain, no gain.
 - No ponga todo el blanquillo en una sola bolsa.

**Objetivos:**
- ¿Qué es la línea de asignación de capital?
- ¿Qué es el radio de Sharpe?
- ¿Cómo deberíamos asignar nuestro capital entre un activo riesgoso y un activo libre de riesgo?

*Referencia:*
- Notas del curso "Portfolio Selection and Risk Management", Rice University, disponible en Coursera.
___

## 1. Línea de asignación de capital

### 1.1. Motivación

El proceso de construcción de un portafolio tiene entonces los siguientes dos pasos:
1. Escoger un portafolio de activos riesgosos.(el mejor)
2. Decidir qué tanto de tu riqueza invertirás en el portafolio y qué tanto invertirás en activos libres de riesgo.

Al paso 2 lo llamamos **decisión de asignación de activos**.

Preguntas importantes:
1. ¿Qué es el portafolio óptimo de activos riesgosos?
 - ¿Cuál es el mejor portafolio de activos riesgosos?
 - Es un portafolio eficiente en media-varianza.
2. ¿Qué es la distribución óptima de activos?
 - ¿Cómo deberíamos distribuir nuestra riqueza entre el portafolo riesgoso óptimo y el activo libre de riesgo?
 - Concepto de **línea de asignación de capital**.
 - Concepto de **radio de Sharpe**.

Dos suposiciones importantes:
- Funciones de utilidad media-varianza.
- Inversionista averso al riesgo.

La idea sorprendente que saldrá de este análisis, es que cualquiera que sea la actitud del inversionista de cara al riesgo, el mejor portafolio de activos riesgosos es idéntico para todos los inversionistas.

Lo que nos importará a cada uno de nosotros en particular, es simplemente la desición óptima de asignación de activos.
___

### 1.2. Línea de asignación de capital

Sean:
- $r_s$ el rendimiento del activo riesgoso,
- $r_f$ el rendimiento libre de riesgo, y
- $w$ la fracción invertida en el activo riesgoso.

 Realizar deducción de la línea de asignación de capital en el tablero.

**Tres doritos después...**

#### Línea de asignación de capital (LAC):
$E[r_p]$ se relaciona con $\sigma_p$ de manera afín. Es decir, mediante la ecuación de una recta:

$$E[r_p]=r_f+\frac{E[r_s-r_f]}{\sigma_s}\sigma_p.$$

- La pendiente de la LAC es el radio de Sharpe $\frac{E[r_s-r_f]}{\sigma_s}=\frac{E[r_s]-r_f}{\sigma_s}$,
- el cual nos dice qué tanto rendimiento obtenemos por unidad de riesgo asumido en la tenencia del activo (portafolio) riesgoso.

Ahora, la pregunta es, ¿dónde sobre esta línea queremos estar?
___

### 1.3. Resolviendo para la asignación óptima de capital

Recapitulando de la clase pasada, tenemos las curvas de indiferencia: **queremos estar en la curva de indiferencia más alta posible, que sea tangente a la LAC**.

 Ver en el tablero.

Analíticamente, el problema es

$$\max_{w} \quad E[U(r_p)]\equiv\max_{w} \quad E[r_p]-\frac{1}{2}\gamma\sigma_p^2,$$

donde los puntos $(\sigma_p,E[r_p])$ se restringen a estar en la LAC, esto es $E[r_p]=r_f+\frac{E[r_s-r_f]}{\sigma_s}\sigma_p$ y $\sigma_p=w\sigma_s$. Entonces el problema anterior se puede escribir de la siguiente manera:

$$\max_{w} \quad r_f+wE[r_s-r_f]-\frac{1}{2}\gamma w^2\sigma_s^2.$$

 Encontrar la $w$ que maximiza la anterior expresión en el tablero.

**Tres doritos después...**

La solución es entonces:

$$w^\ast=\frac{E[r_s-r_f]}{\gamma\sigma_s^2}.$$

De manera intuitiva:
- $w^\ast\propto E[r_s-r_f]$: a más exceso de rendimiento que se obtenga del activo riesgoso, más querremos invertir en él.
- $w^\ast\propto \frac{1}{\gamma}$: mientras más averso al riesgo seas, menos querrás invertir en el activo riesgoso.
- $w^\ast\propto \frac{1}{\sigma_s^2}$: mientras más riesgoso sea el activo, menos querrás invertir en él.
___

## 2. Ejemplo de asignación óptima de capital: acciones y billetes de EU

Pongamos algunos números con algunos datos, para ilustrar la derivación que acabamos de hacer.

En este caso, consideraremos:
- **Portafolio riesgoso**: mercado de acciones de EU (representados en algún índice de mercado como el S&P500).
- **Activo libre de riesgo**: billetes del departamento de tesorería de EU (T-bills).

Tenemos los siguientes datos:

$$E[r_{US}]=11.9\%,\quad \sigma_{US}=19.15\%, \quad r_f=1\%.$$

Recordamos que podemos escribir la expresión de la LAC como:

\begin{align}
E[r_p]&=r_f+\left[\frac{E[r_{US}-r_f]}{\sigma_{US}}\right]\sigma_p\\
      &=0.01+\text{S.R.}\sigma_p,
\end{align}

donde $\text{S.R}=\frac{0.119-0.01}{0.1915}\approx0.569$ es el radio de Sharpe (¿qué es lo que es esto?).

Grafiquemos la LAC con estos datos reales:

```python
# Importamos librerías que vamos a utilizar
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
```

```python
# Datos
Ers= 0.119
ss=0.1915
rf=0.01
# Radio de Sharpe para este activo
RS= (Ers-rf)/ss
# Vector de volatilidades del portafolio (sugerido: 0% a 50%)
sp=np.linspace(0,0.5,100)
# LAC
Erp= rf+RS*sp
RS
```

    0.5691906005221932

```python
# Gráfica
plt.figure(figsize=(8,6))
plt.plot(sp,Erp,lw=2,label='LAC')
plt.plot(0,rf,'or',ms=10,label='risk free')
plt.plot(ss,Ers,'or',ms=10,label='Portafolio riesgoso')
plt.xlabel('Volatilidad $\sigma_p$')
plt.ylabel('Rendimiento Esperado $E[r_p]$')
plt.grid()
plt.legend(loc='best')
plt.show()
```

Bueno, y ¿en qué punto de esta línea querríamos estar?
- Pues ya vimos que depende de tus preferencias.
- En particular, de tu actitud de cara al riesgo, medido por tu coeficiente de aversión al riesgo.

Solución al problema de asignación óptima de capital:

$$\max_{w} \quad E[U(r_p)]$$

$$w^\ast=\frac{E[r_s-r_f]}{\gamma\sigma_s^2}$$

Dado que ya tenemos datos, podemos intentar para varios coeficientes de aversión al riesgo:

```python
# importar pandas
import pandas as pd
```

```python
# Crear un DataFrame con los pesos, rendimiento
# esperado y volatilidad del portafolio óptimo
# entre los activos riesgoso y libre de riesgo
# cuyo índice sean los coeficientes de aversión
# al riesgo del 1 al 10 (enteros)
gamma=np.linspace(1,10,10)
tabla=pd.DataFrame(data={'$\gamma$': gamma,
                        '$w^*$':(Ers-rf)/(gamma*ss**2)})
tabla
```

      $\gamma$
      $w^*$

      0
      1.0
      2.972275

      1
      2.0
      1.486137

      2
      3.0
      0.990758

      3
      4.0
      0.743069

      4
      5.0
      0.594455

      5
      6.0
      0.495379

      6
      7.0
      0.424611

      7
      8.0
      0.371534

      8
      9.0
      0.330253

      9
      10.0
      0.297227

**Interpretacion**

entre mas averso al riesgo menos se va a invertir en portafolios riesgosos.

¿Cómo se interpreta $w^\ast>1$?
- Cuando $0<w^\ast<1$, entonces $0<1-w^\ast<1$. Lo cual implica posiciones largas en el mercado de activos y en el activo libre de riesgo.
- Por el contrario, cuando $w^\ast>1$, tenemos $1-w^\ast<0$. Lo anterior implica una posición corta en el activo libre de riesgo (suponiendo que se puede) y una posición larga (de más del 100%) en el mercado de activos: apalancamiento.

# Anuncios parroquiales.

## 1. Quiz la siguiente clase.

## 2. Tarea 5 segunda entrega para el viernes.

Created with Jupyter by Esteban Jiménez Rodríguez.


# Machine Learning with Graph - Preliminaries part 2
> This notebook covers network properties and some random graph models. Primary source is from Standford CS224W course.
- toc: true
- badges: true
- comments: true
- categories: [cs224w]
- image: images/graph.png

This is my personal notes for [The Standford CS224W Course: Machine Learning with Graphs - Fall 2019](http://web.stanford.edu/class/cs224w/) which I have been studying from its public resources for a couple of months. I will try to keep this blog series live update as my main source knowlege about graph.
The course covers tremendous knowledge for understanding and effectively learning from large-scale network. In general, it is organized into 3 big sections:
- `preliminaries`: provides basic background about networks/graphs.
- `network methods`: methods and algorithms which help us dealing with graph data structure.
- `machine learning with networks`: showing how we are applying machine learning into this field.

This notebook is part 2 of the preliminary section in which we will talk about network properties and some random graph models.

Here is the link to other parts: [1](https://phucnsp.github.io/blog/cs224w/2020/05/27/ML-graph-premilinary-part1.html).

## Network Properties: How to Measure a Network

In this section, we will take a look on four key network properties which can be used to characterize a graph. These properties are extremely useful when we want to compare our model and real-world networks to see whether it fits.
- Degree Distribution: $P(k)$
- Path Length: $h$
- Clustering Coefficient: $C$
- Connected Components: $s$

The definitions will be presented for undirected graphs, but can be easily extended to directed graphs.

### Degree Distribution $P(k)$

The degree distribution $P(k)$ measures the probability that a randomly chosen node has degree $k$.
The degree distribution of a graph $G$ can be summarized by a **normalized histogram**, where we normalize the histogram by the total number of nodes. We can compute the degree distribution of a graph by:
\begin{equation}
P(k)=\frac{N_k}{N}, N_{k}=\#nodes\_with\_degree\_k, N=\#nodes\_in\_graph
\end{equation}

>Note: To extend these definitions to a directed graph, we compute separately both in-degree and out-degree distribution.

### Paths in a Graph

A **path** is a sequence of nodes in which each node is linked to the next one:
\begin{equation}
P_{n}=\left\{i_{0}, i_{1}, i_{2}, \ldots, i_{n}\right\} \\
\end{equation}
such that $\left\{\left(i_{0}, i_{1}\right),\left(i_{1}, i_{2}\right),\left(i_{2}, i_{3}\right), \ldots,\left(i_{n-1}, i_{n}\right)\right\} \in E$

>Note: a path can intersect itself and pass through the same edge multiple times, e.g ACBDCDEG

The **distance(shortest path, geodesic)** between a pair of nodes is defined as the number of edges along the shortest path connecting the nodes. If two nodes are not connected, the distance is usually defined as infinite(or zero).
In a directed graph, paths need to follow the direction of the arrows. Thus, distance is not symmetric for directed graphs. For a graph with weighted edges, the distance is the minimum number of edge weight needed to traverse to get from one node to another.
The **Diameter** of a graph is the maximum(shortest path) distance between any pair of nodes in a graph.
The **average path length** for a connected graph or a strongly connected directed graph is the average shortest path between all connected nodes. We compute the average path length as:
\begin{equation}
\hat{h}=\frac{1}{2E_{\max}} \sum_{i, j \neq i} h_{i j}
\end{equation}
where:
$E_{max}$ is the max number of edges or node pairs; that is, $E_{max}=\left(\begin{array}{c}N \\ 2\end{array}\right)=\frac{N(N-1)}{2}$.
$h_{ij}$ is the distance from node $i$ to node $j$.

>Note: we only compute the average path length over connected pairs of nodes, and thus ignore infinite length paths. In addition, the measure also applied to (strongly) connected components of a graph.

### Clustering Coefficient

The clustering coefficient (for undirected graph) measures what proportion of node $i$'s neighbors are connected. For node $i$ with degree $k_i$, we compute the clustering coeffient as
\begin{equation}
C_i = \frac{2e_i}{k_i(k_i-1)}, C_{i} \in [0,1]
\end{equation}
where:
$e_i$ is the number of edges between the neighbors of node $i$.

$\frac{k_{i}(k_{i}-1)}{2}$ is the maximum number of edges between the $k_i$ neighbors
>Note: the clustering coefficient is undefined for nodes with degree 0 or 1.

We can also compute the average clustering coefficient as
\begin{equation}
C = \frac{1}{N} \sum_{i}^{N} C_{i}.
\end{equation}

The average clustering coefficient allows us to see if edges appear more densely in parts of the network. In social networks, the average clustering coefficient tends to be very high indicating that, as we expect, friends of friends tend to know each other.

### Connectivity

The connectivity of a graph measures the size of the largest connected component. The **largest connected component** is the largest set where any two vertices can be joined by a path.
>Note: Largest component = Giant component

To find connected components:
1. Start from a random node and perform breadth first search (BFS). If you are not familiar wit BFS, see [here.](https://www.hackerearth.com/practice/algorithms/graphs/breadth-first-search/tutorial/)
2. Label the nodes that BFS visits
3. If all the nodes are visited, the network is connected
4. Otherwise find an unvisited node and repeat BFS

### Measure Network Properties of some real-world Networks

Now we will take a look at a real-world network, the protein-protein interation network, and see how much the four key properties are.

Some observations from this network are:
- The degree distribution is highly skewed, in a sense that majority of proteins have small degree and a few proteins have very high degree. On average, a node has around 3 neighbors.
- The diameter of this graph is around 6, it means within 6 steps you can reach any node in the (connected) graph. Does it surprise you? empirically, even with other networks with million of nodes, the diameter is also somewhere lower than 10. This characteristic relates to the concept of depth in graph neural network which we will see later.
- The average clustering is around 0.12, that means, on average, only around 10% of node's neightbors are connected one another.
- Lastly, the largest component has over 80% nodes of the network.

**So are these values "expected"? Are they "surprising"?**
So far we dont know, we need a model to see whether they are suprising or not. In the next part, we will go through some random graph models. Their power to model real-world networks is low but it helps us to have kind of baseline model to compare with other more powerful models. In addition, these random models are simple enough that we can calculate their properties by hand and see if a property can be easily obtained by random.

## Erdos-Renyi Random Graph Model

The Erdos-Renyi Random Graph Model is the simplest model of graphs. This simple model has proven networks properties and is a good baseline to compare real-world graph properties with.
This random graph model comes in two variants:
1. $G_{np}$: undirected graph on $n$ nodes where each edge $(u,v)$ appears i.i.d with probability $p$.
2. $G_{nm}$: undirected graph with $n$ nodes, and $m$ edges picked uniformly at random.

Note that both the $G_{np}$ and $G_{nm}$ graph are not uniquely determined, but rather the result of a random procedure. We can have many different realizations given the same $n$ and $p$

Let's take a look on the key properties of this random graph model.

### Degree Distribution of $G_{np}$

The degree distribution of $G_{np}$ is [binomial](https://www.statisticshowto.com/probability-and-statistics/binomial-theorem/binomial-distribution-formula/). Briefly remind, this distribution can be thought of as simply the probability of a SUCCESS or FAILURE outcome in an experiment that is repeated multiple times. It is a discrete analogue of a Gaussian and has a bell-shape. Binomial distribution has two variable: $n$ - number of times the experiment runs, $p$ - the probability of one specific outcome.

### Clustering Coefficient of $G_{np}$

Recall that the clustering coefficient is computed as:

\begin{equation}
C_{i}=2\frac{e_i}{k_{i}(k_{i}-1)}
\end{equation}

where $e_i$ is the number of edges between $i$'s neighbors. Edges in $G_{np}$ appear i.i.d with probability $p$, so the expected $e_i$ for $G_{np}$ is:

\begin{equation}
\mathbb{E}\left[e_{i}\right]=p \frac{k_{i}\left(k_{i}-1\right)}{2}
\end{equation}

>Note: This is because $\frac{k_i(k_i-1)}{2}$ is the number of distinct pairs of neighbors of node $i$ of degree $k_i$, and each pair is connected with probability $p$.

Thus, the **expected clustering coefficient** is

\begin{equation}
\mathbb{E}\left[C_{i}\right]=\frac{p \cdot k_{i}\left(k_{i}-1\right)}{k_{i}\left(k_{i}-1\right)}=p=\frac{\bar{k}}{n-1} \approx \frac{\bar{k}}{n}
\end{equation}

where $\bar{k}$ is the average degree. From this, we can see that the clustering coefficient of $G_{np}$ is very small because $\bar{k}$ is really small compare to number of nodes $n$. If we generate bigger and bigger graphs with fixed average degree $\bar{k}$, then $C$ decreases with graph size $n$.
\begin{equation}
\mathbb{E}\left[C_{i}\right] \rightarrow 0, n \rightarrow \infty
\end{equation}

### Path Length of $G_{np}$

To discuss the path length of $G_{np}$, we first introduce the concept of **expansion**.
Graph $G(V, E)$ has expansion $\alpha$: if $\forall S \subset V$, the number of edges leaving $S \geq \alpha \cdot \min (|S|,|V \backslash S|)$ (for any subset of nodes, the number of edges leaving is greater than $\alpha$ * the length of subset). Expansion answers the question "if we pick a random set of nodes, how many edges are going to leave the set?". Expansion is a measure of robustness: to disconnect $l$ nodes, one must cut $\geq \alpha \cdot \ell $ edges.
Equivalently, we can say a graph $G(V, E)$ has an expansion $\alpha$ such that:
\begin{equation}
\alpha=\min _{S \subset V} \frac{\# \text { edges leaving } S}{\min (|S|,|V \backslash S|)}
\end{equation}

An important fact about expansion is that in a graph with $n$ nodes with expansion $\alpha$, for all pairs of nodes, there is a pair of path length $O((\log n) / \alpha)$ connecting them. Intuition, we can think of it like if we start with a subset of node and for each step the number of nodes visited increase with a factor of $\alpha$. Thus, it will take $\log n$ of steps to visit all nodes.

For the case of random graph $G_{np}$ and $\log n > np > c$, $c$ is a constant and $np$ is the average degree.
\begin{equation}
diam(G_{np}) = O(\frac{\log n}{\log (np)})
\end{equation}

>Note: the reason we want to have average degree in between logn and a constant c is because we need a reasonable degree to have small diameter.

Thus, we can see that random graphs have good expansion so it takes as logarithmic number of steps for BFS to visit all nodes.
**Key thing to note, the path length of $G_{np}$ is $O(\log n)$. From this result, we can see that $G_{np}$ can grow very large, but nodes will still remain a few hops apart.**

### Connectivity of $G_{np}$

The graphic below shows the evolution of a $G_{np}$ random graph as $p$ change. We can see that there is an emergency of a giant component when average degree higher than 1.
- if $k=1-\epsilon$, then all components are of size $\Omega(\log n)$.
- if $k=1+\epsilon$, there exists 1 component of size $\Omega(n)$, and all other componnents have size $\Omega(\log n)$. In other words, if $\bar{k}>1$, we expect a single large component. In addition, in this case, each node has at least one edge in expectation.

So as average degree $k<1$, fraction of nodes in largest component does not change much but it will start to increase exponentially as $k>1$.

### Comparing a real network with $G_{np}$

In order to see if our random model fits well with a real network, let's take a look on **MSN Messenger** network in 1 month of activify which has 180 million users and 1.3 billion edges.

As we can see, giant connected component and average path length might fit between them but clustering coefficient and degree distribution do not. The problem with the random network model are:
- Degree distribution differs from that of real networks
- Giant component in most real networks does NOT emerge through a phase transition
- No local structure - clustering coefficient is too low.
Thus, we can conclude that real networks obviously are not random.

Although the random network $G_{np}$ is WRONG to model real network but it is still useful because:
- It is the reference model for the rest of the class
- It will help us calculate many quantities, that can then be compared to the real data
- It will help us understand to what degree a particular property is the result of some random process.

## The Small-World Model

This model is the improvement of random model in which it tackles the problem of constructing graph with low average path length and high clustering.
The Small-World model was first introduced in 1998 by Ducan J.Watts and Steven Strogatz. To create such model, we employ the following steps:
1. Start with a low-dimensional regular lattice(ring) by connecting each node to $k$ neighbors on its right and $k$ neightbors on its left, with $k \geq 2$. So we have a lattice with high clustering coefficient.
2. Rewire each edge with probability $p$ by moving its endpoint to a randomly chosen node. This step will add/remove edges to create shortcuts to join remote parts of the lattice. So we are introducing randomness (shortcut) into the lattice. The rewiring allows us to "interpolate" between a regular lattice and a random graph.
>Note: Clustering implies edge "locality". Randomness enables "shortcuts"

Then, we make the following observations:
- At $p=0$ where no rewiring has occured, this remains a grid network with high clustering, high diameter.
- For $0<p<1$ some edges have been rewired, but most of the structure remains. This implies both *locality* and **shortcuts**. This allows for both high clustering and low diameter.
- At $p=1$ where all edges have been randomly rewired, this is a Erdos-Renyi (ER) random graph with low clustering, low diameter.
>Note: Small world model are parameterized by the probability of rewiring $p \in[0,1]$

From a social network perspective, the phenomenon of introducing shortcuts while keeping strong structure is intuitive. While most our friends are local, but we also have a few long distance friendships in different countries which is enough to collapse the diamter of the human social network, explaining the popular notion of **Six Degrees of Separation".**
In summary, the small world model:
- provides insight on the interplay between clustering and the small-world.
- captures the structure of many realistic networks.
- accounts for the high clustering of real networks.
- but it does not lead to the correct degree distribution of real-world network.

## Kronecker Graph Model

Models of graph generation have been studied extensively. Such models allow us to generate graphs for simulations and hypothesis testing when collecting the real graph is difficult, and also forces us to examine the network properties that generative models should obey to be considered realistic.
In formulating graph generation models, there are two important considerations:
- The ability to generate realistic networks.
- The mathematical tractability of the models, which allows for the rigorous analysis of network properties.

The Kronecker Graph Model is a **recursive graph generation model** that combines both mathematical tractability and realistic static and temporal network properties. The intuition underlying the Kronecker Graph Model is **self-similarity**, where the whole has the same shape as one or more of its parts.

### Kronecker Product

The **Kronecker product $\otimes$**, a non-standard matrix operation, is a way of generating self-similar matrics. For two arbitarily sized matrices $\mathbf{A} \in \mathbb{R}^{n \times m}$ and $\mathbf{B} \in \mathbb{R}^{k \times l}, \mathbf{A} \otimes \mathbf{B} \in \mathbb{R}^{n k \times m l}$, their Kronecker product is:
\begin{equation}
\mathbf{C}={\mathbf{A}} \otimes \mathbf{B} \doteq\left(\begin{array}{cccc}
a_{1,1} \mathbf{B} & a_{1,2} \mathbf{B} & \dots & a_{1, m} \mathbf{B} \\
a_{2,1} \mathbf{B} & a_{2,2} \mathbf{B} & \dots & a_{2, m} \mathbf{B} \\
\vdots & \vdots & \ddots & \vdots \\
a_{n, 1} \mathbf{B} & a_{n, 2} \mathbf{B} & \dots & a_{n, m} \mathbf{B}
\end{array}\right)
\end{equation}

For example, we have that
$$\begin{bmatrix} 1&2\\3&4 \end{bmatrix} \otimes \begin{bmatrix} 0&5\\6&7 \end{bmatrix}
= \begin{bmatrix} 1  \begin{bmatrix} 0&5\\6&7 \end{bmatrix}  &2  \begin{bmatrix} 0&5\\6&7 \end{bmatrix}  \\3  \begin{bmatrix} 0&5\\6&7 \end{bmatrix}  &4  \begin{bmatrix} 0&5\\6&7 \end{bmatrix}  \end{bmatrix}
= \begin{bmatrix}
1 \times 0 & 1 \times 5 & 2 \times 0 &  2 \times 5\\
1 \times 6 & 1 \times 7 & 2 \times 6 &  2 \times 7\\
3 \times 0 & 3 \times 5 & 4 \times 0 &  4 \times 5\\
3 \times 6 & 3 \times 7 & 4 \times 6 &  4 \times 7
\end{bmatrix}
= \begin{bmatrix}
0 & 5 & 0 &  10\\
6 & 7 & 12 &  14\\
0 & 15 & 0 &  20\\
18 & 21 & 24 &  28
\end{bmatrix}$$

### Kronecker Graphs

To use the Kronecker product in graph generation, we define the **Kronecker product of two graphs** as the Kronecker product of the adjacency matrices of the two graphs.
**Kronecker graph** is obtained by growing sequence of graphs by iterating the Kronecker product over the initiator matrix $K_1$ (an adjacency matrix of a graph):
$$K_1^{[m]}=K_m = \underbrace{K_1 \otimes K_1 \otimes \dots K_1}_{\text{m times}}=  K_{m-1} \otimes K_1$$

Intuitively, the Kronecker power construction can be imagined as recursive growth of the communities within the graph, with nodes in the community recursively getting expanded into miniature copies of the community.

One can easily use multiple initiator matrices $(K_{1}^{\prime}, K_{1}^{\prime \prime}, K_{1}^{\prime \prime \prime})$ (even of different sizes), which iteratively affects the structure of the larger graph.

### Stochastic Kronecker Graphs

Up to now, we have only considered $K_1$ initiator matrices with binary values $\{0,1\}$. However, such graphs generated from such initiator matrices have "staircase" effects in the degree distributions and individual values occur very frequently because of the discrete nature of $K_1$.
To negate this effect, stochasticity is introduced by relaxing the assumption that the entries in the initiator matrix can only take binary values. Instead entries in $\Theta_1$ can take values on the interval $[0, 1]$, and each represents the probability of that particular edge appearing. Then the matrix (and all the generated larger matrix products) represents the probability distribution over all possible graphs from that matrix.

More concretely, for probability matrix $\Theta_1$, we compute the $k^{th}$ Kronecker power $\Theta_k$ as the large stochastic adjacency matrix. Each entry $p_{uv}$ in $\Theta_k$ then represents the probability of edge $(u, v)$ appearing.

>Notes: the probabilities do not have to sum up to 1 as each the probability of each edge appearing is independent from other edges.

To obtain an instance of a graph, we then sample from the distribution by sampling each edge with probability given by the corresponding entry in the stochastic adjacency matrix. The sampling can be thought of as the outcomes of flipping biased coins where the bias is parameterized from each entry in the matrix.

However, this means that the time to naively generate an instance is quadratic in the size of the graph, $O(N^2)$! too slow.
Is there a faster way?

### Fast Generation of Stochastic Kronecker Graphs

A fast heuristic procedure that takes time linear in the number of edges to generate a graph exists.
The general idea can be described as follows: for each edge, we recursively choose sub-regions of the large stochastic matrix with probability proportional to $p_{uv} \in \Theta_{1}$ until we descend to a single cell of the large stochastic matrix. We replace the edge there. For a Kronecker graph of $k^{th}$ power, $\Theta_k$, the descent will take $k$ steps.
For example, we consider the case where $\Theta_1$ is a $2 x 2$ matrix, such that
\begin{equation}
\Theta=\left[\begin{array}{ll}
a & b \\
c & d
\end{array}\right]
\end{equation}

For graph $G$ with $n = 2^k$ nodes, we need to define how many edges we want to generate (we can reference of the ratio between number of nodes and number of edges). Then, the algorithm to generate each edge is shown below:

In practice, the stochastic Kronecker graph model is able to generate graphs that match the properties of real world networks well. To read more about the Kronecker Graph models, refer to `J Leskovec et al., Kronecker Graphs: An Approach to Modeling Networks (2010)`.

## References
- main source: [Standford CS224W Machine Learning with Graphs](http://web.stanford.edu/class/cs224w/)
- other sources
    - [Graph on structured documents blog post](https://towardsdatascience.com/using-graph-convolutional-neural-networks-on-structured-documents-for-information-extraction-c1088dcd2b8f)
    - [Graph theory blog post](https://towardsdatascience.com/graph-theory-and-deep-learning-know-hows-6556b0e9891b)
    - [Graph embedding blog post](https://towardsdatascience.com/overview-of-deep-learning-on-graph-embeddings-4305c10ad4a4)
    - [Graph convolution blog post](https://towardsdatascience.com/graph-convolutional-networks-for-geometric-deep-learning-1faf17dee008)
    - [Geometric deep learning blog post](https://medium.com/@flawnsontong1/what-is-geometric-deep-learning-b2adb662d91d)
    - review papers about graphs: [1](https://arxiv.org/pdf/1611.08097.pdf), [2](https://arxiv.org/pdf/1901.00596.pdf)
    - Pytorch Geometric:[1](https://pytorch-geometric.readthedocs.io/en/latest/notes/introduction.html), [2](https://towardsdatascience.com/hands-on-graph-neural-networks-with-pytorch-pytorch-geometric-359487e221a8)

```python

```


# Chris Daigle
## Convex Optimization w/ Python
## Exam

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sympy import *
```

```python
df = pd.read_csv(
    '/Users/daiglechris/Git/ConvexOptimizationWithPython/Data/Stocks2-closeP.csv',
    sep = ",")
```

```python
df.head()
```

      timestamp
      F_close
      BA_close
      RTN_close
      VLO_close
      AAL_close
      CI_close
      IBM_close
      PG_close

      0
      2018-12-03
      9.60
      359.96
      172.87
      81.30
      39.65
      224.84
      125.31
      93.32

      1
      2018-11-30
      9.41
      346.76
      175.34
      79.90
      40.16
      223.38
      124.27
      94.51

      2
      2018-11-29
      9.37
      342.56
      173.58
      79.47
      38.42
      222.52
      121.48
      92.82

      3
      2018-11-28
      9.41
      333.50
      173.61
      79.42
      38.94
      221.72
      123.00
      93.01

      4
      2018-11-27
      9.28
      318.03
      171.67
      77.43
      38.29
      216.80
      120.03
      92.72

### Question 1:
(a) Compute the one day returns $\left (\dfrac{p_{t+1}-p_{t}}{p_{t}} \right )$ of each stock and find the mean and the variance of each stock's returns.

```python
oneDayReturns = df.iloc[:,1:].apply(lambda x: ((x - (x.shift(1))) / (x.shift(1))))
names = list(oneDayReturns.columns)
meansOfReturns = pd.DataFrame(oneDayReturns.describe().iloc[1].values,
                              index = names)
varianceOfReturns = oneDayReturns.var()
table = pd.concat([meansOfReturns, varianceOfReturns],
                  axis = 1)
table.columns = ['Mean of the Returns', 'Variance of the Returns']
table
```

      Mean of the Returns
      Variance of the Returns

      F_close
      0.001533
      0.000352

      BA_close
      -0.000098
      0.000331

      RTN_close
      0.001566
      0.000214

      VLO_close
      0.003054
      0.000501

      AAL_close
      -0.000345
      0.000649

      CI_close
      -0.002542
      0.000194

      IBM_close
      0.001668
      0.000267

      PG_close
      -0.001568
      0.000145

|**Name of Stock**| **Mean of the Returns** | **Variance of the Returns**    |
|------------|:--------------------------------------------:|--------------:|
| Ford       | 0.001533| 0.000352  |
| Boeing     | 0.000098| 0.000331 |
| Raytheon       | 0.001566| 0.000214 |
| Valero Energy     | 0.003054 | 0.000501 |
| American Airlines       | -0.000345 | 0.000649  |
| Cigna     |-0.002542| 0.000194|
| IBM       | 0.001668| 0.000267 |
| Procter & Gamble     | -0.001568|0.000145|

(b) Plot the returns of the stocks with the two highest variance and the stocks with the two highest mean. Save a pdf copy of you graphs with “plt.savefig(ʼsample.pdfʼ). Send the pdf file to olivier.morand@uconn.edu with “graph-your name” in the title of your email.

```python
highestVar = table.sort_values(by = 'Variance of the Returns',
                               axis = 0,
                               ascending=False).head(2)
highestMean = table.sort_values(by = 'Mean of the Returns',
                                axis = 0,
                                ascending=False).head(2)
dfHighest = pd.concat([highestMean, highestVar])
dfReturnHighest = oneDayReturns[dfHighest.T.columns]
```

```python
dfReturnHighest.plot(grid = True,
                     figsize = (12,8)).axhline(y = 0,
                                               color = "black",
                                               lw = 2)
plt.title('Returns of the stocks with the two highest variance and the stocks with the two highest mean')
plt.savefig('graph-Daigle.pdf')
plt.show()
```

### Question 2:
Two clients, John and Jane, have asked you for help to build a portfolio in which to invest their income. You seek a combination of the 8 stocks that will maximize each clientʼs utility. A portfolio is a vector $X = (x_{1}, ..., x_{8})$ with $\forall i = 1, ... 8$, $0 \le x_{i} \le 1$ and $\sum_{i=1}^{8} x_{i} = 1$ where $x_{i}$ is the fraction of income to be spend on asset $i$. Both clients have utility:
$$X.\overline{R}^{T} - \frac{1}{2}AX\Sigma X^{T}$$
where $\overline{R}$ is the $1 \times 8$ vector of means calculated in 1, and $\Sigma$ is the $8 \times 8$ covariance matrix of returns. John is risk averse and his risk coefficient is $A = 2$ while Jane's coefficient is $A = 10$.

(a) Explain below why the problem is convex.

The problem is convex because the Hessian Matrix of $X\Sigma X^{T}$ is positive semidefinite (PSD), $\frac{1}{2} A$ only scales the value of $X\Sigma X^{T}$, and the first portion $X.\overline{R}^{T}$ is just a vector of scalars, singleton or otherwise. So, we only need to evaluate the Hessian of $X\Sigma X^{T}$.

```python
x1, x2, x3, x4, x5, x6, x7, x8 = symbols('ford, Boeing, Raytheon, Valero, AmericanAir, Cigna, IBM, PG')
X = np.array([x1, x2, x3, x4, x5, x6, x7, x8])
R = meansOfReturns
bigSigma = oneDayReturns.cov()
Z = np.array(X*bigSigma*X.T)
```

```python
def hessian(x):
    """
    Calculate the hessian matrix with finite differences
    Parameters:
       - x : ndarray
    Returns:
       an array of shape (x.dim, x.ndim) + x.shape
       where the array[i, j, ...] corresponds to the second derivative x_ij
    """
    x_grad = np.gradient(x)
    hessian = np.empty((x.ndim, x.ndim) + x.shape, dtype=x.dtype)
    for k, grad_k in enumerate(x_grad):
        # iterate over dimensions
        # apply gradient again to every component of the first derivative.
        tmp_grad = np.gradient(grad_k)
        for l, grad_kl in enumerate(tmp_grad):
            hessian[k, l, :, :] = grad_kl
    return hessian
```

```python
H = hessian(np.array(bigSigma))
np.linalg.det(H)
```

    array([[ 2.20799925e-63, -6.76560985e-48],
           [-7.95272270e-48, -3.57667386e-65]])

This works out to a series of zeros in the matrix. I'm not sure why there are 4 values, but regardless, zero times any real value is zero and thus, the matrix is PSD and the problem is thus convex.

```python

```


```python
import sympy as sym
import matplotlib.pyplot as plt
%matplotlib inline
sym.init_printing()
```

# Caracterización de Sistemas en el Dominio del Tiempo

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Respuesta al impulso

A continuación se discutirá sobre la respuesta $y(t)$ de un sistema $\mathcal{H}$, Lineal en Invariante en el Tiempo (LTI), ante una señal de entrada $x(t)$.

Tenga en cuenta que la señal de entrada puede representarse como una integral:

\begin{equation}
x(t) = \int_{-\infty}^{\infty} x(\tau) \cdot \delta(t-\tau) \; d \tau
\end{equation}

Si se introduce esta representación de la señal de entrada en la transformación que realiza el sistema $\mathcal{H}$ para formar la salida  $y(t) = \mathcal{H} \{ x(t) \}$, se obtiene:

\begin{equation}
y(t) = \mathcal{H} \left\{ \int_{-\infty}^{\infty} x(\tau) \cdot \delta(t-\tau) \; d \tau \right\}
\end{equation}

donde $\mathcal{H} \{ \cdot \}$ representa un operador que encuentra la salida del sistema. Como $x(\tau)$ puede verse como una constante ante la variable $t$, la expresión para $y(t)$ puede escribirse como:

\begin{equation}
y(t) = \int_{-\infty}^{\infty}  x(\tau) \cdot \mathcal{H} \left\{ \delta(t-\tau) \right\} \; d \tau
\end{equation}

Observe que $\mathcal{H} \{\cdot\}$ opera únicamente sobre el impulso de Dirac.

La respuesta de un sistema ante una señal impulso de Dirac es conocida como la
[*respuesta al impulso*](https://en.wikipedia.org/wiki/Impulse_response).

\begin{equation}
h(t) = \mathcal{H} \left\{ \delta(t) \right\}
\end{equation}

Si el sistema es invariante en el tiempo, la respuesta a un impulso desplazado es $\mathcal{H} \left\{ \delta(t-\tau) \right\} = h(t-\tau)$. Así, para un sistema LTI se obtiene que:

\begin{equation}
y(t) = \int_{-\infty}^{\infty}  x(\tau) \cdot h(t-\tau) \; d \tau
\end{equation}

Esta operación es conocida como [*convolución*](https://en.wikipedia.org/wiki/Convolution). Se simboliza mediante $*$, de manera que la integral se represente como $y(t) = x(t) * h(t)$ o, para algunos autores, $y(t) = (x*h)(t)$.

La respuesta $y(t)$ de un sistema LTI \mathcal{H} ante una entrada $x(t)$ se determina completamente a partir de su respuesta impulsional $h(t)$ como la convolución de la entrada $x(t)$ con la respuesta impulsional $h(t)$.

Un sistema LTI con $x(t)$ como entrada y $y(t)$ como salida es descrito mediante la siguiente Ecuación Diferencial Ordinaria (ODE):

\begin{equation}
y(t) + \frac{d}{dt} y(t) = x(t)
\end{equation}

Se desea encontrar la respuesta del sistema ante una entrada $x(t) = e^{- 2 t} \cdot \epsilon(t)$

### Solución A
Primero, se va a resolver la ODE teniendo en cuenta  que deben satisfacerse las condiciones iniciales $y(t)\big\vert_{t = 0-} = 0$ y $\frac{d}{dt}y(t)\big\vert_{t = 0-} = 0$ relacionadas con la causalidad.

a. Defina la ODE en`SymPy`

```python
t = sym.symbols('t', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(y + y.diff(t), x)
ode
```

Se resuelve la ODE para la entrada establecida y considerando que las constantes de integración deben garantiza las condiciones inicales.

```python
xinput = sym.exp(-2*t)*sym.Heaviside(t)

solution = sym.dsolve(ode.subs(x, xinput))
solution
```

```python

integration_constants = sym.solve(
    (solution.rhs.limit(t, 0, '-'), solution.rhs.diff(t).limit(t, 0, '-')), 'C1')

y1 = solution.subs(integration_constants)
y1
```

Se grafica la señal de salida obtenida al resolver la ODE.

```python
plt.rcParams['figure.figsize'] = 7, 2
sym.plot(y1.rhs, (t,-1,10), ylabel=r'$y(t)$');
```

### Solución B
Ahora se va a resolver el problema a través de la respuesta impulsional.

La respuesta impulsional $h(t)$ se encuentra al resolver la ODE para una señal impulso de Dirac como entrada, $x(t) = \delta(t)$.

```python
h = sym.Function('h')(t)

solution2 = sym.dsolve(ode.subs(x, sym.DiracDelta(t)).subs(y, h))

integration_constants = sym.solve((solution2.rhs.limit(
    t, 0, '-'), solution2.rhs.diff(t).limit(t, 0, '-')), 'C1')

h = solution2.subs(integration_constants)
h
```

La respuesta impulsional se grafica.

```python
plt.rcParams['figure.figsize'] = 7, 2
sym.plot(h.rhs, (t,-1,10), ylabel=r'$h(t)$');
```

Ahora se calcula la convolución $y(t) = x(t) * h(t)$ como:

\begin{equation}
y(t) = \int_{-\infty}^{\infty}  x(\tau) \cdot h(t-\tau) \; d \tau
\end{equation}

Debido a que $h(t)$ y $x(t)$ son señales causales, la expresión de convolución se convierte en:

\begin{equation}
y(t) = \int_{0}^{t} x(\tau) \cdot h(t - \tau) \; d\tau \; \forall t \geq 0
\end{equation}

Observe que $y(t) = 0 \; \forall t<0$.

```python
tau = sym.symbols('tau', real=True)

y2 = sym.integrate(xinput.subs(t, tau) * h.rhs.subs(t, t-tau), (tau, 0, t))
y2
```

Se grafica la señal de salida obtenida mediante la convolución.

```python
plt.rcParams['figure.figsize'] = 7, 2
sym.plot(y2, (t,-1,10), ylabel=r'$y(t)$');
```

Las señales obtenidas con los dos métodos deben ser iguales.  Verifiquemos las dos señales.

```python
y1
```

```python
y2
```

Al comparar las dos señales graficamente se obtiene:

```python
plt.rcParams['figure.figsize'] = 7, 2
graf1 = sym.plot(y1.rhs, (t,-1,10), ylabel=r'$y(t)$', show=False, line_color='red',legend=True, label = 'ODE')
graf2 = sym.plot(y2, (t,-1,10), show=False, legend=True, label = 'convolucion')
graf1.extend(graf2)
graf1.show()
```

.ipynb_checkpoints/3-1 LTI Respuesta impulsional-checkpoint.ipynb