# Music Machine Learning - Bayesian inference

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. An introduction to [Bayesian inference](#bayesian)
2. A formal introduction to [Variational Auto-Encoders](#vae) (VAEs)
3. An explanation of the [implementation](#implem) of VAEs
4. Some [modifications and tips to improve the reconstruction](#improve) of VAEs **(exercise)**

<a id="recap"> </a>

## Introduction on Bayesian inference

Here, we discuss *Bayesian inference* and how to use the **Bayes theorem** to perform classification. First, we will see how to derive *estimators* for the different properties of the distributions, and verify that these are *unbiased*. Then, we will implement the **Maximum Likelihood** Estimators (MLE) in order to perform classification of a dataset. First, we will assess the case where parameters are known to implement the discriminant function and decision rule. Then, we will perform the MLE to obtain the means and covariance matrix for each class.

To understand these concepts graphically, we will rely on both the `scikit-learn` and `Pytorch` libraries (with the `probability` package).

```python
import torch
import torch.distributions as distribution
import torch.distributions.transforms as transform
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from helper_plot import hdr_plot_style, plot_gaussian_ellipsoid
hdr_plot_style()
```

### Bayesian framework

Two alternative interpretations of probability can be considered

  * **Frequentist** (*classical* approach), assumes that probability is the *long-term frequency of events*. This becomes harder to interpret when events have no long-term frequency (eg. probability in an election, which happens only once). In that case, frequentists consider the *frequency of occurrences across alternative realities*, which defines the probability.
  * **Bayesian** interprets probability as measure of *believability in an event*. Therefore, a probability is measure of *belief*, or confidence, of an event occurring. This definition leaves room for conflicting beliefs based on the different *information* about the world. Hence, bayesian inference is mostly based on *updating your beliefs* after considering new *evidence*.

To align with probability notation, we denote a belief about event $a$ as $p\left(a\right)$, called the *prior probability* of an event to occur. We denote the updated belief as $ p\left( a \mid e \right) $, interpreted as the probability of $a$ *given* the new evidence $e$, called the *posterior probability*. The prior belief is not completely removed, but we *re-weight this prior* to incorporate new evidence $e$ (i.e. we put more weight, or confidence, on some beliefs versus others). By introducing prior uncertainty about events, we admit that any guess we make can be wrong. As we gather an *infinite* amount of evidence $N \rightarrow \infty$, the Bayesian results (often) align with frequentist results. Hence for small $N$, inference is *unstable*, where frequentist estimates have more variance and larger confidence intervals. However, by introducing a prior, and returning probabilities, we *preserve the uncertainty* that reflects the instability on a small $N$ dataset.

Updating the *prior belief* to obtain our *posterior belief* is done via the the Bayes' Theorem

$$
\begin{equation}
 p\left( a \mid e \right) \propto \frac{ p\left(e \mid a\right) p\left(a\right) } {p\left(e\right) }
\end{equation}
$$

We see that our posterior belief of event $a$ given the new evidence $e$ is proportional to ($\propto$) the *likelihood* of observing this particular evidence $e$ given the event $a$ ($p\left(e \mid a\right)$) multiplied by our prior belief in that particular event $a$ ($p\left(a\right)$).

### Using Bayesian classification (in `scikit-learn`)

The Bayesian classification methods rely on Bayes's theorem, where we are interested in finding the probability of a label $y$ given some observed features, which we can write as $p(y~|~{\rm features})$. Bayes's theorem tells us how to express this in terms of quantities that we can compute more directly

$$
p(y~|~{\rm features}) = \frac{p({\rm features}~|~y)p(y)}{p({\rm features})}
$$

If we are trying to decide between two labels ($y_1$ and $y_2$), then one way to make this decision is to compute the ratio of the posterior probabilities for each label

$$
\frac{p(y_1~|~{\rm features})}{p(y_2~|~{\rm features})} = \frac{p({\rm features}~|~y_1)}{p({\rm features}~|~y_2)}\frac{p(y_1)}{p(y_2)}
$$

All we need now is some model by which we can compute $p({\rm features}~|~y_i)$ for each label.
Such a model is called a *generative model* because it specifies the hypothetical random process that generates the data.
Specifying this generative model for each label is the main piece of the training of such a Bayesian classifier.
The general version of such a training step is a very difficult task, but we can make it simpler through the use of some simplifying assumptions about the form of this model. If we make *very naive assumptions* (called *Naive Bayes*) about the generative model for each label, we can find a rough approximation of the generative model for each class, and then proceed with the Bayesian classification.

#### Gaussian Naive Bayes

Perhaps the easiest naive Bayes classifier to understand is Gaussian naive Bayes. In this classifier, the assumption is that *data from each label is drawn from a simple Gaussian distribution*. Imagine that we have the following data

```python
hdr_plot_style()
from sklearn.datasets import make_blobs
X, y = make_blobs(200, 2, centers=2, random_state=2, cluster_std=2.2)
plt.figure(figsize=(10, 8)); plt.scatter(X[:, 0], X[:, 1], c=y, s=80, cmap='RdBu', edgecolor='w'); plt.grid(True)
```

One extremely fast way to create a simple model is to assume that the data is described by a Gaussian distribution with no covariance between dimensions. This model can be fit by simply finding the mean and standard deviation of the points within each label, which is all you need to define such a distribution.

With this generative model for each class, we have a simple recipe to compute the likelihood $P({\rm features}~|~L_1)$ for any data point, and thus we can quickly compute the posterior ratio and determine which label is the most probable for a given point. This procedure is implemented in Scikit-Learn's ``sklearn.naive_bayes.GaussianNB`` estimator

```python
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
model.fit(X, y);
```

Now let's generate some new data and predict the label:

```python
rng = np.random.RandomState(0)
Xnew = [-6, -14] + [14, 18] * rng.rand(2000, 2)
ynew = model.predict(Xnew)
```

Now we can plot the data and see where the decision boundary is

```python
fig = plt.figure(figsize=(10,8)); ax = fig.add_subplot(111)
# predict the classification probabilities on a grid
xlim = X[:, 0].min() - .5, X[:, 0].max() + .5
ylim = X[:, 1].min() - .5, X[:, 1].max() + .5
xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 71), np.linspace(ylim[0], ylim[1], 81))
Z = model.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]
Z = Z.reshape(xx.shape)
ax.contourf(xx, yy, Z, cmap=plt.cm.RdBu, alpha=.8)
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu', edgecolor='w');
```

We see a slightly curved boundary in the classifications (generally, the boundary in Gaussian naive Bayes is quadratic). A nice piece of this Bayesian formalism is that it naturally allows for probabilistic classification, which we can compute using the ``predict_proba`` method:

```python
yprob = model.predict_proba(Xnew)
yprob[-8:].round(2)
```

    array([[0.9 , 0.1 ],
           [1.  , 0.  ],
           [1.  , 0.  ],
           [1.  , 0.  ],
           [0.98, 0.02],
           [1.  , 0.  ],
           [0.  , 1.  ],
           [0.95, 0.05]])

The columns give the posterior probabilities of the first and second label, respectively. If you are looking for estimates of uncertainty in your classification, Bayesian approaches like this can be a useful approach. Of course, the final classification will only be as good as the model assumptions that lead to it, which is why Gaussian naive Bayes often does not produce very good results.

## Bayesian inference

Suppose we have coin and want to estimate the probability of heads ($p$) for it. The coin is Bernoulli distributed:

$$
\begin{equation}
\phi(x)= p^x (1-p)^{(1-x)}
\end{equation}
$$

where $x$ is the outcome, *1* for heads and *0* for tails. Based on $n$ *independent* flips, we have the likelihood:

$$
\begin{equation}
\mathcal{L}(p|\mathbf{x})= \prod_{i=1}^n p^{ x_i }(1-p)^{1-x_i}
\end{equation}
$$

(the independent-trials assumption allows us to just substitute everything into $ \phi(x)$).

The idea of *maximum likelihood* will be to maximize this as the function of $p$ after given all of the $x_i$ data. This means that our estimator, $\hat{p}$, is a function of the observed $x_i$ data, and as such, is a random variable with its own distribution.

**Defining the estimator**

The only way to know for sure that our estimator is correctly defined is to check if the estimator is unbiased, namely, if

$$ \mathbb{E}(\hat{p}) = p $$

### Understanding this behavior with `pytorch`

Without solving for the previous exercise, we can  Let's say that we have 100 samples from a Bernoulli distribution.

```python
bernoulli = distribution.Bernoulli(0.2)
sample = bernoulli.sample((100, ))
sns.distplot(sample)
plt.title("Samples from a Bernoulli (p = .2)")
torch.mean(sample)
```

Now we can use our estimator, and define our problem more formally, by defining our input samples as a dataset of observations $x$, and we are trying to model this set, as a `Bernoulli` distribution, for which the $p$ parameter is unknown (here defined as a `Variable`).

```python
from torch.autograd import Variable
x = Variable(sample).type(torch.FloatTensor)
p = Variable(torch.rand(1), requires_grad=True)
```

Now we can use our estimator to gradually compute the Maximum Likelihood, in order to uncover the estimated probability of the underlying distribution, solely based on observing the samples.

```python
learning_rate = 0.00002
for t in range(1000):
    NLL = -torch.sum(torch.log(x*p + (1-x)*(1-p)) )
    NLL.backward()
    if t % 200 == 0:
        print("loglik = %.4f - p = %.4f - dL/dp = %.4f"%(NLL.data.numpy(), p.data.numpy(), p.grad.data.numpy()))
    p.data -= learning_rate * p.grad.data
    p.grad.data.zero_()
print('Final probability p =', p.data[0])
```

    loglik = 104.9821 - p = 0.7100 - dL/dp = 242.8272
    loglik = 53.4532 - p = 0.2998 - dL/dp = 42.7879
    loglik = 51.4267 - p = 0.2203 - dL/dp = 5.9784
    loglik = 51.3959 - p = 0.2109 - dL/dp = 0.5635
    loglik = 51.3957 - p = 0.2101 - dL/dp = 0.0501
    Final probability p = tensor(0.2100)

***

**Exercise**

  1. Compute the *log-likelihood* $J=\log(\mathcal{L}(p \mid \mathbf{x}))$ of our given problem
  2. Based on this, compute its derivative $ \frac{dJ}{dp} $
  3. Solve it to find the estimator $\hat{p}$
  4. Verify that this estimator is unbiased $ \mathbb{E}(\hat{p}) = p $
  5. Compute the variance of the estimator $ \mathbb{E}\left(\hat{p}^2\right) $

***

```python
x = Variable(sample).type(torch.FloatTensor)
p = torch.rand(1)

######################
# YOUR CODE GOES HERE
######################

```

### Full estimator density

In general, computing the mean and variance of the estimator is insufficient to characterize the underlying probability density of $\hat{p}$, except if we knew that $\hat{p}$ were normally distributed. This is where the [*central limit theorem*](http://mathworld.wolfram.com/CentralLimitTheorem.html). Indeed, the form of the estimator, implies that $\hat{p}$ is normally distributed, but only *asymptotically*, which doesn't quantify how many samples $n$ we need. Unfortunately, in the real world, each sample may be precious. Hence, to write out the full density for $\hat{p}$, we first have to ask what is the probability that the estimator will equal a specific value such as

$$
\begin{equation}
\hat{p} = \frac{1}{n}\sum_{i=1}^n x_i  = 0
\end{equation}
$$

This can only happen when $x_i=0$,  $\forall i$. The corresponding probability can be computed from the density

$$
\begin{equation}
f(\mathbf{x},p)= \prod_{i=1}^n \left(p^{x_i} (1-p)^{1-x_i}  \right)
\end{equation}
$$

$$
\begin{equation}
f\left(\sum_{i=1}^n x_i  = 0,p\right)= \left(1-p\right)^n
\end{equation}
$$

Likewise, if $\lbrace x_i \rbrace$ has one $i^{th}$ value equal to one, then

$$
\begin{equation}
f\left(\sum_{i=1}^n x_i  = 1,p\right)= n p \prod_{i=1}^{n-1} \left(1-p\right)
\end{equation}
$$

where the $n$ comes from the $n$ ways to pick one value equal to one from the $n$ elements $x_i$. Continuing this way, we can construct the entire density as

$$
\begin{equation}
f\left(\sum_{i=1}^n x_i  = k,p\right)= \binom{n}{k} p^k  (1-p)^{n-k}
\end{equation}
$$

where the term on the left is the binomial coefficient of $n$ things taken $k$ at a time. This is the binomial distribution and it's not the density for $\hat{p}$, but rather for $n\hat{p}$. We'll leave this as-is because it's easier to work with below. We just have to remember to keep track of the $n$ factor.

## Maximum Likelihood implementation

Maximum Likelihood Estimate (MLE) allows to perform typical statistical pattern classification tasks. In the cases where **probabilistic models and parameters are known**, the design of a Bayes' classifier is rather easy. However, in real applications, we are rarely given this information and this is where the MLE comes into play.

MLE still **requires partial knowledge** about the problem. We have to assume that the **model of the class conditional densities is known** (usually Gaussian distributions). Hence, Using MLE, we want to estimate the values of the parameters of a given distribution for the class-conditional densities, for example, the *mean* and *variance* assuming that the class-conditional densities are *normal*  distributed (Gaussian) with

$$
\begin{equation}
p(\pmb x   \mid   y_i) \sim N(\mu, \sigma^2)
\end{equation}
$$

### Parameters known

Imagine that we want to classify data consisting of two-dimensional patterns, $\pmb{x} = [x_1, x_2] \in \mathbb{R}^{2}$ that could belong to 1 out of 3 classes $y_1,y_2,y_3$.

Let's assume the following information about the model where we use continuous univariate normal (Gaussian) model for the class-conditional densities

$$
\begin{equation}
p(\pmb x \mid y_j) \sim N(\pmb \mu \mid \Sigma) = \frac{1}{(2\pi)^{d/2}   \mid \Sigma|^{1/2}} exp \bigg[ -\frac{1}{2}(\pmb x - \pmb \mu)^t \Sigma^{-1}(\pmb x - \pmb \mu) \bigg]
\end{equation}
$$

Furthermore, we consider for this first problem that we know the distributions of the classes, ie. their mean and covariances.

$$
\begin{equation}
p([x_1, x_2]^t \mid y_1) ∼ N([0,0],3I), \\
p([x_1, x_2]^t \mid y_2) ∼ N([9,0],3I), \\
p([x_1, x_2]^t \mid y_3) ∼ N([6,6],4I),
\end{equation}
$$

Therefore, the means of the sample distributions for 2-dimensional features are defined as

$$
\begin{equation}
\pmb{\mu}_{\,1} = \bigg[ 0, 0 \bigg],    \pmb{\mu}_{\,2} = \bigg[ 9,  0 \bigg],    \pmb{\mu}_{\,3} = \bigg[ 6,  6 \bigg]
\end{equation}
$$

The **covariance matrices** for the statistically independent and identically distributed ('i.i.d') features

$$
\begin{array}{ccc}
\Sigma_1 = \bigg[
\begin{array}{cc}
3 & 0\\
0 & 3 \\
\end{array} \bigg],
\Sigma_2 = \bigg[
\begin{array}{cc}
3 & 0\\
0 & 3 \\
\end{array} \bigg],
\Sigma_3 = \bigg[
\begin{array}{cc}
4 & 0\\
0 & 4 \\
\end{array} \bigg]
\end{array}$$

Finally, we consider that all classes have an **equal prior probability**

$$p(y_1)   =    p(y_2)   =   p(y_3)   = \frac{1}{3}$$

***

**Exercise**

  1. Generate some data (samples from the multivariate Gaussians) following classes distributions
  2. Plot the class-dependent data

***

```python
# 7.0 - Generate data with known parameters

nb_patterns = 100
# Generate random patterns for class 1
mu1 = np.array([0,0])
cov1 = np.array([[1,0],[0,1]])
# Generate random patterns for class 2
mu2 = np.array([9,0])
cov2 = np.array([[3,0],[0,3]])
# Generate random patterns for class 3
mu3 = np.array([6,6])
cov3 = np.array([[7,0],[0,7]])

######################
# YOUR CODE GOES HERE
######################

# Prepare concatenated versions of class properties
mu_vals = {};
mu_vals[0] = np.array([mu1]).transpose()
mu_vals[1] = np.array([mu2]).transpose()
mu_vals[2] = np.array([mu3]).transpose()
cov_vals = {}
cov_vals[0] = cov1
cov_vals[1] = cov2
cov_vals[2] = cov3
# Plot the corresponding data
plt.figure(figsize=(12, 8))
plt.scatter(x1samples[:, 0], x1samples[:, 1], s=60, marker='o', c=[0, 0.8, 0], edgecolors='w')
plt.scatter(x2samples[:, 0], x2samples[:, 1], s=60, marker='o', c=[0, 0, 0.8], edgecolors='w')
plt.scatter(x3samples[:, 0], x3samples[:, 1], s=60, marker='o', c=[0.8, 0, 0], edgecolors='w')
h = plot_gaussian_ellipsoid(mu1, cov1, 2, color=[0, 0.8, 0]);
h = plot_gaussian_ellipsoid(mu2, cov2, 2, color=[0, 0, 0.8]);
h = plot_gaussian_ellipsoid(mu3, cov3, 2, color=[0.8, 0, 0]);
plt.title('Training Dataset')
plt.ylabel('x2')
plt.xlabel('x1')
```

Here, our **objective function** is to maximize the discriminant function $g_i(\pmb x)$, which we define as the posterior probability to perform a **minimum-error classification** (Bayes classifier).

$$
\begin{equation}
g_1(\pmb x) = p(y_1 \mid   \pmb{x}), \quad  g_2(\pmb{x}) = p(y_2 \mid   \pmb{x}), \quad  g_3(\pmb{x}) = p(y_3 \mid   \pmb{x})
\end{equation}
$$

So that our decision rule is to choose the class $\omega_i$ for which $g_i(\pmb x)$ is max., where

 $$
 \begin{equation}
 \quad g_i(\pmb{x}) = \pmb{x}^{\,t} \bigg( - \frac{1}{2} \Sigma_i^{-1} \bigg) \pmb{x} + \bigg( \Sigma_i^{-1} \pmb{\mu}_{\,i}\bigg)^t \pmb x + \bigg( -\frac{1}{2} \pmb{\mu}_{\,i}^{\,t}  \Sigma_{i}^{-1} \pmb{\mu}_{\,i} -\frac{1}{2} ln(\left|\Sigma_i\right|)\bigg)
 \end{equation}
 $$

***

**Exercise**

  1. Implement the discriminant function
  2. Implement the decision rule (classifier)
  3. Classify the data generated in the previous exercise
  4. Plot the confusion matrix
  5. Calculate the empirical error

***

```python
import operator

def discriminant_function(x_vec, mu_vec, cov_mat):
    # Calculates the value of the discriminant function for a dx1 dimensional
    # sample given the covariance matrix and mean vector.
    #
    #   x_vec: A dx1 dimensional numpy array representing the sample.
    #   cov_mat: numpy array of the covariance matrix.
    #   mu_vec: dx1 dimensional numpy array of the sample mean.
    #
    #   Returns a float value g as result of the discriminant function.

    ######################
    # YOUR CODE GOES HERE
    ######################

    return float(g)

def classify_data(x_vec, g, mu_vecs, cov_mats):
    #    Classifies an input sample into 1 out of 3 classes determined by
    #    maximizing the discriminant function g_i().

    #    Keyword arguments:
    #        x_vec: A dx1 dimensional numpy array representing the sample.
    #        g: The discriminant function.
    #        mu_vecs: A list of mean vectors as input for g.
    #        cov_mats: A list of covariance matrices as input for g.
    #
    #    Returns the max probability and class id.

    ######################
    # YOUR CODE GOES HERE
    ######################

    return maxVal, maxId

x1classes = np.zeros((nb_patterns, 1))
conf_matrix = np.zeros((3, 3))
for i in range(nb_patterns):
    x, g = classify_data(x1samples[i, :], discriminant_function, mu_vals, cov_vals);
    x1classes[i] = g;
    conf_matrix[0, g] = conf_matrix[0, g] + 1;
x2classes = np.zeros((nb_patterns, 1))
for i in range(nb_patterns):
    x, g = classify_data(x2samples[i, :], discriminant_function, mu_vals, cov_vals);
    x2classes[i] = g;
    conf_matrix[1, g] = conf_matrix[1, g] + 1;
x3classes = np.zeros((nb_patterns, 1))
for i in range(nb_patterns):
    x, g = classify_data(x3samples[i, :], discriminant_function, mu_vals, cov_vals);
    x3classes[i] = g;
    conf_matrix[2, g] = conf_matrix[2, g] + 1;

print('%16s \t %s \t %s \t %s\n' % (' ', 'class 1', 'class 2', 'class 3'));
print('%16s \t %f \t %f \t %f\n' % ('class 1', conf_matrix[0, 0], conf_matrix[0, 1], conf_matrix[0, 2]));
print('%16s \t %f \t %f \t %f\n' % ('class 2', conf_matrix[1, 0], conf_matrix[1, 1], conf_matrix[1, 2]));
print('%16s \t %f \t %f \t %f\n' % ('class 3', conf_matrix[2, 0], conf_matrix[2, 1], conf_matrix[2, 2]));
```

                     	 class 1 	 class 2 	 class 3

             class 1 	 90.000000 	 0.000000 	 10.000000

             class 2 	 0.000000 	 86.000000 	 14.000000

             class 3 	 0.000000 	 2.000000 	 98.000000

### Unknown parameters

In contrast to the previous case, let us assume that we only know the number of parameters for the class conditional densities $p (\pmb x \mid  y_i)$, and we want to use a Maximum Likelihood Estimation (MLE) to estimate the quantities of these parameters from the training data.

Given the information about the our model (the data is normal distributed) the 2 parameters to be estimated for each class are $\pmb \mu_i$ and $\pmb \Sigma_i$, which are summarized by the parameter vector

$$
\begin{equation}
\pmb \theta_i = \bigg[ \begin{array}{c}
\ \theta_{i1} \\
\ \theta_{i2} \\
\end{array} \bigg]=
\bigg[ \begin{array}{c}
\pmb \mu_i \\
\pmb \Sigma_i \\
\end{array} \bigg]
\end{equation}
$$

For the Maximum Likelihood Estimate (MLE), we assume that we have a set of samples $D = \left\{ \pmb x_1, \pmb x_2,..., \pmb x_n \right\} $ that are *i.i.d.* (independent and identically distributed, drawn with probability $p(\pmb x \mid   y_i, \pmb \theta_i) )$.  Thus, we can **work with each class separately** and omit the class labels, so that we write the probability density as $p(\pmb x \mid \pmb \theta)$

**Likelihood of $ \pmb \theta $**

Thus, the probability of observing $D = \left\{ \pmb x_1, \pmb x_2,..., \pmb x_n \right\} $ is

$$
\begin{equation}
p(D  \mid    \pmb \theta ) = p(\pmb x_1   \mid   \pmb \theta )  \cdot   p(\pmb x_2   \mid  \pmb \theta )   \cdot  ...    p(\pmb x_n   \mid   \pmb \theta ) = \prod_{k=1}^{n}   p(\pmb x_k \pmb   \mid   \pmb \theta  )
\end{equation}
$$

Where $p(D  \mid    \pmb  \theta )$ is also called the ***likelihood of $\pmb\ \theta$***

We know that $p([x_1,x_2]^t)  ∼   N(\pmb \mu,\pmb \Sigma) $ (remember that we dropped the class labels, since we are working with every class separately). And the mutlivariate normal density is given as

$$
\begin{equation}
\quad \quad p(\pmb x) = \frac{1}{(2\pi)^{d/2}   |\Sigma|^{1/2}} exp \bigg[ -\frac{1}{2}(\pmb x - \pmb \mu)^t \Sigma^{-1}(\pmb x - \pmb \mu) \bigg]
\end{equation}
$$

Therefore, we obtain

$$
\begin{equation}
p(D  \mid    \pmb \theta ) = \prod_{k=1}^{n}   p(\pmb x_k \pmb   \mid   \pmb \theta  ) =  \prod_{k=1}^{n}   \frac{1}{(2\pi)^{d/2}   |\Sigma|^{1/2}} exp \bigg[ -\frac{1}{2}(\pmb x - \pmb \mu)^t \Sigma^{-1}(\pmb x - \pmb \mu) \bigg]
\end{equation}
$$
and the log of the multivariate density

$$
\begin{equation}
l(\pmb\theta) =  \sum\limits_{k=1}^{n} - \frac{1}{2}(\pmb x - \pmb \mu)^t \pmb \Sigma^{-1}   (\pmb x - \pmb \mu) - \frac{d}{2}   ln   2\pi - \frac{1}{2}  ln   |\pmb\Sigma|
\end{equation}
$$

In order to obtain the MLE $\boldsymbol{\hat{\theta}}$, we maximize $l (\pmb  \theta)$, which can be done via differentiation

$$
\begin{equation}
\nabla_{\pmb \theta} \equiv \begin{bmatrix}
\frac{\partial   }{\partial   \theta_1} \\
\frac{\partial   }{\partial   \theta_2}
\end{bmatrix} = \begin{bmatrix}
\frac{\partial   }{\partial   \pmb \mu} \\
\frac{\partial   }{\partial   \pmb \sigma}
\end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
\nabla_{\pmb \theta} l = \sum\limits_{k=1}^n \nabla_{\pmb \theta}  ln  p(\pmb x| \pmb \theta) = 0
\end{equation}
$$

***

**Exercise**
<div markdown = "1">

  1. Perform the differentiation for $\frac{\partial   \mathcal{L}}{\partial   \pmb \mu}$ to obtain the estimator of the mean $\hat{\mu}$
  2. Perform the differentiation for $\frac{\partial   \mathcal{L}}{\partial   \pmb \Sigma}$ to obtain the estimator of the covariance matrix $\hat{\Sigma}$
  3. Implement the two estimators as functions based on a set of data
  4. Apply these estimators (MLE) in order to obtain estimated parameters
  5. Re-compute the classification errors on the previous dataset

***

```python
########################################################
# YOUR CODE GOES HERE (Perform mu estimates)
########################################################

# mu_est_1 = ?
# mu_est_2 = ?
# mu_est_3 = ?

print('%16s \t %s \t %s \t %s \t %s \t %s \t %s' % ('', 'mu1_1   ', 'mu1_2  ', 'mu2_1  ', 'mu2_2  ', 'mu3_1  ', 'mu3_2  '));
print('%16s \t %f \t %f \t %f \t %f \t %f \t %f' % ('MLE', mu_est_1[0], mu_est_1[1], mu_est_2[0], mu_est_2[1], mu_est_3[0], mu_est_3[1]));
print('%16s \t %f \t %f \t %f \t %f \t %f \t %f' % ('Truth', mu1[0], mu1[1], mu2[0], mu2[1], mu3[0], mu3[1]));

def mle_covariance(x_samples, mu_est):
    #    Calculates the Maximum Likelihood Estimate for the covariance matrix.
    #
    #    Keyword Arguments:
    #        x_samples: np.array of the samples for 1 class, n x d dimensional
    #        mu_est: np.array of the mean MLE, d x 1 dimensional
    #
    #    Returns the MLE for the covariance matrix as d x d numpy array.

    ######################
    # YOUR CODE GOES HERE
    ######################

    return cov_est_

cov_est_1 = mle_covariance(x1samples, np.array([mu_est_1]).transpose());
cov_est_2 = mle_covariance(x2samples, np.array([mu_est_2]).transpose());
cov_est_3 = mle_covariance(x3samples, np.array([mu_est_3]).transpose());

print('%16s \t %s \t %s \t %s \t %s \t %s \t %s' % ('', 'cov1_1   ', 'cov1_2  ', 'cov2_1  ', 'cov2_2  ', 'cov3_1  ', 'cov3_2  '));
print('%16s \t %f \t %f \t %f \t %f \t %f \t %f' % ('MLE', cov_est_1[0, 0], cov_est_1[1, 0], cov_est_2[0, 0], cov_est_2[1, 0], cov_est_3[0, 0], cov_est_3[1, 0]));
print('%16s \t %f \t %f \t %f \t %f \t %f \t %f' % ('Truth', cov1[0, 0], cov1[1, 0], cov2[0, 0], cov2[1, 0], cov3[0, 0], cov3[1, 0]));

mu_estimates = {};
mu_estimates[0] = np.array([mu_est_1]).transpose()
mu_estimates[1] = np.array([mu_est_2]).transpose()
mu_estimates[2] = np.array([mu_est_3]).transpose()
cov_estimates = {};
cov_estimates[0] = cov_est_1;
cov_estimates[1] = cov_est_2;
cov_estimates[2] = cov_est_3;

# Plot the corresponding data
plt.figure(figsize=(12, 8))
plt.scatter(x1samples[:,0], x1samples[:,1], s=40, marker='o', c=[0, 0.8, 0], edgecolors='w');
plt.scatter(x2samples[:,0], x2samples[:,1], s=40, marker='s', c=[0, 0, 0.8], edgecolors='w');
plt.scatter(x3samples[:,0], x3samples[:,1], s=40, marker='^', c=[0.8, 0, 0], edgecolors='w');
h = plot_gaussian_ellipsoid(mu1, cov1, 2, color=[0.2, 0.6, 0.2]);
h = plot_gaussian_ellipsoid(mu2, cov2, 2, color=[0.2, 0.2, 0.6]);
h = plot_gaussian_ellipsoid(mu3, cov3, 2, color=[0.6, 0.2, 0.2]);
h = plot_gaussian_ellipsoid(mu_est_1, cov_est_1, 2, color=[0.1, 0.99, 0.1]);
h = plot_gaussian_ellipsoid(mu_est_2, cov_est_2, 2, color=[0.1, 0.1, 0.99]);
h = plot_gaussian_ellipsoid(mu_est_3, cov_est_3, 2, color=[0.99, 0.1, 0.1]);
plt.title('Comparing estimated MLE Gaussians');
```

## Audio source separation

The maximum likelihood estimator (MLE) is widely used in practical signal modeling and we can show that the MLE is equivalent to the least squares estimator for a wide class of problems, including well resolved sinusoids in white noise. We are going to consider a model consisting of a complex sinusoid in additive white (complex) noise:

$$
\displaystyle x(n) = {\cal A}e^{j\omega_{0} n} + v(n)
$$

Here, $ {\cal A}= A e^{j\phi} $ is the complex amplitude of the sinusoid, and $ v(n) $ is white noise that we assume to be Gaussian distributed with zero mean. Hence, we assume that its probability density function is given by

$$
\displaystyle p_{v}(\nu) = \frac{1}{\pi \sigma_{v}^2} e^{-\frac{\vert\nu\vert^2}{\sigma_{v}^2}}.
$$

We express the zero-mean Gaussian assumption by writing

$$
\displaystyle v(n) \sim {\cal N}(0,\sigma_{v}^2)
$$

The parameter  $ \sigma_{v}^2 $ is the *variance* of the random process $ v(n) $ , and $ \sigma_{v} $ is its standard deviation. It turns out that when Gaussian random variables $ v(n) $ are uncorrelated (i.e., when $ v(n) $ is white noise), they are also independent. This means that the probability of observing particular values of $ v(n) $ and $ v(m) $ is given by the product of their respective probabilities. We will now use this fact to compute an explicit probability for observing any data sequence $ x(n) $

Since the sinusoidal part of our signal model,  $ {\cal A}e^{j\omega_{0}n}$ , is deterministic; i.e., it does not including any random components; it may be treated as the time-varying mean of a Gaussian random process $ x(n) $ . That is, our signal model can be rewritten as

$$
\displaystyle x(n) \sim {\cal N}({\cal A}e^{j\omega_{0} n},\sigma_{v}^2)
$$

and the probability density function for the whole set of observations $ x(n) $ ,  $ n=0,1,2,\ldots,N-1 $ is given by

$$
\displaystyle p(x) = p[x(0)] p[x(1)]\cdots p[x(N-1)] = \left(\frac{1}{\pi \sigma_v^2}\right)^N e^{-\frac{1}{\sigma_v^2}\sum_{n=0}^{N-1} \left\vert x(n) - {\cal A}e^{j\omega_0 n}\right\vert^2}
$$

Thus, given the noise variance  $ \sigma_v^2 $ and the three sinusoidal parameters $ A,\phi,\omega_0 $ (remember that $ {\cal A}= A e^{j\phi} $ ), we can compute the relative probability of any observed data samples $ x(n) $ .

We can generalize this approach in order to perform a complete blind audio source separation algorithm, such as detailed in the following paper

Févotte, C., & Cardoso, J. F. "Maximum likelihood approach for blind audio source separation using time-frequency Gaussian source models". *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics*, 2005. (pp. 78-81). IEEE.

You can try to implement this by relying on the [full paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.640.6981&rep=rep1&type=pdf) that details this method.

***

**Exercise**
<div markdown = "1">

  1. Implement the single sinusoid extraction
  2. Apply this approach to multiple sinusoids
  3. Follow the paper to implement blind source separation

***

```python

```


```python
# import Point, Polygon
from sympy import Point, Polygon, Line

# creating points using Point()
p1, p2, p3, p4 = map(Point, [(0, 0), (1, 0), (5, 1), (0, 1)])

# creating polygon using Polygon()
poly1 = Polygon(p1, p2, p3, p4)

# using intersection()
isIntersection = poly1.intersection(Line(p1, Point(3, 2)))

print(isIntersection)
```

    [Point2D(0, 0), Point2D(3/2, 1)]

```python

```


# Mecánica con SymPy

_Si SymPy te ha parecido hasta ahora un CAS decente e incluso interesante (nada como tener los resultados en $\LaTeX$ incrustados en el notebook y la sintaxis de Python para hacer cálculo simbólico) entonces espera a ver el paquete `mechanics`. Con él, podremos manipular velocidades y aceleraciones de sólidos expresadas en distintos sistemas de referencia con una facilidad impresionante._

_Tienes disponible la documentación de `mechanics` en http://docs.sympy.org/0.7.5/modules/physics/mechanics/index.html._

## Sistemas de referencia

El objeto primordial que vamos a manejar van a ser los sistemas de referencia. Podremos definir relaciones geométricas entre ellos y de esta forma las transformaciones de vectores entre un sistema y otro serán triviales.

La manera usual de empezar a trabajar con SymPy es importar la función `init_session`:
```
from sympy import init_session
init_session(use_latex=True)```

 Esta función ya se encarga de importar todas las funciones básicas y preparar las salidas gráficas. Sin embargo, en este momento, esta función se encuentra en mantenimiento para su uso dentro de los notebooks por lo que activaremos la salida gráfica e importaremos las funciones de la manera usual. Puedes consultar el estado de la corrección en: https://github.com/sympy/sympy/pull/13300 y https://github.com/sympy/sympy/issues/13319 .

```python
from sympy import init_printing
init_printing()
```

```python
from sympy import symbols
```

Todo lo que necesitamos está en `sympy.physics.mechanics`, incluyendo la clase `ReferenceFrame`. Nada más crear un sistema de referencia podemos acceder a sus versores unitarios: `x`, `y` y `z`.

http://docs.sympy.org/0.7.5/modules/physics/vector/vectors.html

```python
from sympy.physics.mechanics import ReferenceFrame
```

```python
A = ReferenceFrame("A")
A.x
```

Y para definir vectores solo tenemos que **multiplicar cada componente por su versor**:

```python
2 * A.x - 1 * A.y
```

De ahora en adelante, para trabajar como si nos enfrentáramos a un problema de la escuela, vamos a hacer dos cosas:

* Definir un sistema inercial $1$ del que partir, para así poder referir todos los demás sistemas a él.
* Que los versores de ese sistema sean $i, j, k$.

```python
A = ReferenceFrame("1", latexs=['\mathbf{i}', '\mathbf{j}', '\mathbf{k}'])
A.x + A.y + A.z
```

Y para no tener que hacerlo siempre, un pequeño truco de magia:

```python
# Definimos nuestra propia clase para que los versores sean IJK
# aeropython: preserve
class IJKReferenceFrame(ReferenceFrame):
    def __init__(self, name):
        super().__init__(name, latexs=['\mathbf{%s}_{%s}' % (idx, name) for idx in ("i", "j", "k")])
        self.i = self.x
        self.j = self.y
        self.k = self.z
```

```python
A = IJKReferenceFrame("1")
A.i + A.j + A.k
```

### Álgebra vectorial

Nuestros vectores funcionan también con símbolos, y podemos realizar las operaciones de producto escalar y producto vectorial con ellos.

```python
R, V = symbols('R, V', positive=True)
r1 = R * (A.x + A.y + A.z)
v1 = V * (A.x - 2 * A.z)
```

```python
r1
```

```python
v1
```

```python
from sympy.physics.mechanics import dot, cross
```

```python
r1.dot(v1)
dot(r1, v1)
r1 & v1
```

```python
r1.cross(v1)
cross(r1, v1)
r1 ^ v1
```

Podemos hallar también la norma de los vectores con su método `magnitude` e incluso normalizarlos con `normalize`:

```python
(r1 ^ v1).magnitude()
```

```python
(r1 ^ v1).normalize()
```

##### Ejercicio

Usando directamente la fórmula para la derivada en ejes móviles:

$$\left(\frac{\operatorname{d}\!\mathbf{a}}{\operatorname{d}\!t}\right)_1 = \left(\frac{\operatorname{d}\!\mathbf{a}}{\operatorname{d}\!t}\right)_0 + \mathbf{\omega}_{01}\! \times \mathbf{a}$$

Calcula la derivada del vector de posición $R \mathbf{i}_0$, siendo $A_0$ un sistema de referencia que gira respecto al inercial con velocidad angular $\mathbf{\omega}_{01}=\Omega \mathbf{k}_0$. **¿Cuál es el módulo de la derivada?**

```python
R, Omega = symbols('R, Omega', positive=True)
A0 = IJKReferenceFrame('0')
```

```python
a = R * A0.i
a
```

```python
omega01 = Omega * A0.k
omega01
```

```python
da = omega01 ^ a
da
```

```python
da.magnitude()
```

Si no especificaste `positive=True` vas a ver algo como $\sqrt{\Omega^2 R^2}$. Debería haber una forma de simplificar esta expresión _a posteriori_, pero de momento no funciona del todo bien. Preparando este notebook nos hemos dado cuenta y ya les hemos avisado :) https://github.com/sympy/sympy/issues/8326

### Movimiento relativo

¿A quién no le gusta multiplicar matrices de rotación? Para esa minoría que lo detesta, existe SymPy. Para ello debemos especificar la orientación de nuestros sistemas de referencia usando el método `orient`, y recuperaremos la matriz de cosenos directores usando el método `dcm`.

```python
A1 = IJKReferenceFrame("1")
A0 = IJKReferenceFrame("0")
```

```python
phi = symbols('phi')
A0.orient(A1, 'Axis', [phi, A1.z])  # Rotación phi alrededor del eje A1.z
A0.dcm(A1)  # "Direct Cosine Matrix"
```

$$\left[\begin{matrix}\cos{\left (\phi \right )} & \sin{\left (\phi \right )} & 0\\- \sin{\left (\phi \right )} & \cos{\left (\phi \right )} & 0\\0 & 0 & 1\end{matrix}\right]$$

Usando el argumento `Axis` hemos especificado que rotamos el sistema un ángulo especificado alrededor de un eje. Otros métodos son:

* `Body`: se especifican los tres ángulos de Euler.
* `Space`: igual que `Body`, pero las rotaciones se aplican en orden inverso.
* `Quaternion`: utilizando cuaternios, rotación alrededor de un vector unitario $\lambda$ una cantidad $\theta$.

¿Qué es lo bueno de usar uno de estos métodos? ¡Que **siempre** tenemos la transformación bien definida! Es imposible meter "a capón" una matriz de rotación que sea incorrecta o absurda.

#### Diferente sistema de referencia

Para expresar un vector en otro sistema de referencia, no hay más que usar los métodos `express` o `to_matrix`:

```python
A0.x.express(A1)
```

```python
A0.x.to_matrix(A1)
```

$$\left[\begin{matrix}\cos{\left (\phi \right )}\\\sin{\left (\phi \right )}\\0\end{matrix}\right]$$

#### Símbolos dinámicos

Si queremos especificar que un símbolo puede variar con el tiempo, hay que usar la función `dynamicsymbols`:

```python
from sympy.physics.mechanics import dynamicsymbols
```

```python
alpha = dynamicsymbols('alpha')
alpha
```

Y pedir su derivada con el método `diff`:

```python
alpha.diff()
```

##### Ejercicio

(Sacado de Cuerva et al. "Teoría de los Helicópteros")

**Obtener la matriz de rotación de la pala $B$ respecto a los ejes $A1$.**

```python
A = IJKReferenceFrame("A")
```

```python
A1 = IJKReferenceFrame("A1")
psi = dynamicsymbols('psi')
A1.orient(A, 'Axis', [psi, A.z])
A1.dcm(A)  # T_{A1A}
```

$$\left[\begin{matrix}\cos{\left (\psi{\left (t \right )} \right )} & \sin{\left (\psi{\left (t \right )} \right )} & 0\\- \sin{\left (\psi{\left (t \right )} \right )} & \cos{\left (\psi{\left (t \right )} \right )} & 0\\0 & 0 & 1\end{matrix}\right]$$

```python
A2 = IJKReferenceFrame("A2")
beta = dynamicsymbols('beta')
A2.orient(A1, 'Axis', [beta, -A1.y])
A2.dcm(A1)  # T_{A2A1}
```

$$\left[\begin{matrix}\cos{\left (\beta{\left (t \right )} \right )} & 0 & \sin{\left (\beta{\left (t \right )} \right )}\\0 & 1 & 0\\- \sin{\left (\beta{\left (t \right )} \right )} & 0 & \cos{\left (\beta{\left (t \right )} \right )}\end{matrix}\right]$$

```python
A3 = IJKReferenceFrame("A3")
zeta = dynamicsymbols('zeta')
A3.orient(A2, 'Axis', [zeta, A2.z])
A3.dcm(A1)  # T_{A3A1}
```

$$\left[\begin{matrix}\cos{\left (\beta{\left (t \right )} \right )} \cos{\left (\zeta\left(t\right) \right )} & \sin{\left (\zeta\left(t\right) \right )} & \sin{\left (\beta{\left (t \right )} \right )} \cos{\left (\zeta\left(t\right) \right )}\\- \sin{\left (\zeta\left(t\right) \right )} \cos{\left (\beta{\left (t \right )} \right )} & \cos{\left (\zeta\left(t\right) \right )} & - \sin{\left (\beta{\left (t \right )} \right )} \sin{\left (\zeta\left(t\right) \right )}\\- \sin{\left (\beta{\left (t \right )} \right )} & 0 & \cos{\left (\beta{\left (t \right )} \right )}\end{matrix}\right]$$

```python
B = IJKReferenceFrame("B")
theta = dynamicsymbols('theta')
B.orient(A3, 'Axis', [theta, A3.x])
B.dcm(A3)  # T_{BA3}
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & \cos{\left (\theta{\left (t \right )} \right )} & \sin{\left (\theta{\left (t \right )} \right )}\\0 & - \sin{\left (\theta{\left (t \right )} \right )} & \cos{\left (\theta{\left (t \right )} \right )}\end{matrix}\right]$$

```python
B.dcm(A2)
```

$$\left[\begin{matrix}\cos{\left (\zeta\left(t\right) \right )} & \sin{\left (\zeta\left(t\right) \right )} & 0\\- \sin{\left (\zeta\left(t\right) \right )} \cos{\left (\theta{\left (t \right )} \right )} & \cos{\left (\theta{\left (t \right )} \right )} \cos{\left (\zeta\left(t\right) \right )} & \sin{\left (\theta{\left (t \right )} \right )}\\\sin{\left (\theta{\left (t \right )} \right )} \sin{\left (\zeta\left(t\right) \right )} & - \sin{\left (\theta{\left (t \right )} \right )} \cos{\left (\zeta\left(t\right) \right )} & \cos{\left (\theta{\left (t \right )} \right )}\end{matrix}\right]$$

```python
B.dcm(A1)
```

$$\left[\begin{matrix}\cos{\left (\beta{\left (t \right )} \right )} \cos{\left (\zeta\left(t\right) \right )} & \sin{\left (\zeta\left(t\right) \right )} & \sin{\left (\beta{\left (t \right )} \right )} \cos{\left (\zeta\left(t\right) \right )}\\- \sin{\left (\beta{\left (t \right )} \right )} \sin{\left (\theta{\left (t \right )} \right )} - \sin{\left (\zeta\left(t\right) \right )} \cos{\left (\beta{\left (t \right )} \right )} \cos{\left (\theta{\left (t \right )} \right )} & \cos{\left (\theta{\left (t \right )} \right )} \cos{\left (\zeta\left(t\right) \right )} & - \sin{\left (\beta{\left (t \right )} \right )} \sin{\left (\zeta\left(t\right) \right )} \cos{\left (\theta{\left (t \right )} \right )} + \sin{\left (\theta{\left (t \right )} \right )} \cos{\left (\beta{\left (t \right )} \right )}\\- \sin{\left (\beta{\left (t \right )} \right )} \cos{\left (\theta{\left (t \right )} \right )} + \sin{\left (\theta{\left (t \right )} \right )} \sin{\left (\zeta\left(t\right) \right )} \cos{\left (\beta{\left (t \right )} \right )} & - \sin{\left (\theta{\left (t \right )} \right )} \cos{\left (\zeta\left(t\right) \right )} & \sin{\left (\beta{\left (t \right )} \right )} \sin{\left (\theta{\left (t \right )} \right )} \sin{\left (\zeta\left(t\right) \right )} + \cos{\left (\beta{\left (t \right )} \right )} \cos{\left (\theta{\left (t \right )} \right )}\end{matrix}\right]$$

#### Velocidad angular

También podemos hallar la velocidad angular de un sistema respecto a otro usando el método `ang_vel_in`:

```python
B.ang_vel_in(A2)
```

```python
B.ang_vel_in(A)
```

```python
B.ang_vel_in(A).express(A)
```

En ocasiones, la representación gráfica puede fallar, pero se puede volver a desactivar y activar llamando a la función`init_printing(pretty_print=True)` con diferentes valores (True/False) para `pretty_print`

### Derivada en ejes móviles

Hacer una derivada con la fórmula lo hace cualquiera, pero SymPy puede encargarse automáticamente.

```python
v1 = A1.x
v1
```

```python
#v1.diff(dynamicsymbols._t, A2)
dv1 = v1.diff(symbols('t'), A)
dv1
```

```python
dv1.to_matrix(A1)
```

$$\left[\begin{matrix}0\\\sin^{2}{\left (\psi{\left (t \right )} \right )} \frac{d}{d t} \psi{\left (t \right )} + \cos^{2}{\left (\psi{\left (t \right )} \right )} \frac{d}{d t} \psi{\left (t \right )}\\0\end{matrix}\right]$$

```python
(dv1 & A1.j).simplify()
```

### Puntos, velocidades y la rueda que no desliza

El último paso que nos queda para completar la cinemática es la posibilidad de definir puntos en sólidos y aplicar su campo de velocidades. SymPy también permite esto, y para ello no tenemos más que importar la clase `Point`.

```python
from sympy.physics.mechanics import Point
```

```python
O = Point("O")
```

Para trabajar como lo haríamos en la escuela, vamos a especificar que $O$ es el origen de $A$, y para eso vamos a imponer que su velocidad es cero con el método `set_vel`:

```python
O.set_vel(A, 0)
```

Para definir nuevos puntos, podemos utilizar el método `locate_new`:

```python
e_b = symbols('e_b')
E_b = O.locatenew('E_b', e_b * A1.x)
```

Y para obtener vectores de un punto a otro, el método `pos_from`:

```python
E_b.pos_from(O)
```

La notación de este paquete está influenciada por el libro Kane, T. R. & Levinson, D. A. "Dynamics, Theory and Applications". Es ligeramente distinto a como estudiamos nosotros en la escuela, pero ¡están abiertos a que les hagamos cualquier tipo de sugerencia! https://github.com/sympy/sympy/issues/2584#issuecomment-31552654

Por último, el **campo de velocidades de un sólido rígido** se formula usando el método `v2pt_theory`.

$$v^P_A = v^O_A + \omega_{A_1 A} \times \mathbf{OP}$$

Este método pertenece *al punto del cual queremos conocer la velocidad* y recibe tres parámetros:

* `O`, punto de velocidad conocida respecto a A
* `A`, sistema de referencia donde queremos calcular la velocidad
* `A1`, sistema de referencia donde están fijos ambos puntos (_sistema de arrastre_)

Por tanto, para hallar la velocidad del punto que acabamos de crear:

```python
E_b.v2pt_theory(O, A, A1)
```

##### Ejercicio

(Apuntes de Óscar López Rebollal)

**¡Halla la velocidad y la aceleración de $P$!**

```python
# Creamos nuestros sistemas de referencia
A1 = IJKReferenceFrame('1')
A0 = IJKReferenceFrame('0')
A2 = IJKReferenceFrame('2')
```

```python
# Creamos los símbolos dinámicos necesarios
xi, theta = dynamicsymbols('xi, theta')
xi, theta
```

```python
# Orientamos los sistemas de referencia
A0.orient(A1, 'Axis', [0, A1.k])  # A0 no gira respecto a A1
A2.orient(A0, 'Axis', [theta, A0.k])
```

```python
A2.dcm(A1)
```

$$\left[\begin{matrix}\cos{\left (\theta{\left (t \right )} \right )} & \sin{\left (\theta{\left (t \right )} \right )} & 0\\- \sin{\left (\theta{\left (t \right )} \right )} & \cos{\left (\theta{\left (t \right )} \right )} & 0\\0 & 0 & 1\end{matrix}\right]$$

```python
# Creamos el punto C, centro del disco, y especificamos su velocidad
# respecto a A1
C = Point('C')
C.set_vel(A1, xi.diff() * A1.x)
```

```python
# Localizamos el punto P, punto fijo del disco, respecto a C, en
# el sistema A2 (que gira solidariamente con el disco)
R = symbols('R')
P = C.locatenew('P', -R * A2.j)
P.pos_from(C)
```

```python
# Hallamos la velocidad de P en A1, expresada en A0
# ¡Con esta llamada ya estamos diciendo que C y P son fijos en A2!
P.v2pt_theory(C, A1, A2).express(A0)
```

**Misión cumplida :)**

---

_Hemos hecho un repaso bastante profundo de las posibilidades del paquete `mechanics` de SymPy. Nos hemos dejado algunas cosas en el tintero pero no demasiadas: esta funcionalidad aún se está expandiendo y necesita pulir algunos detalles._

**Referencias**

* Capítulo de **aeromecánica** del libro de Cuerva y otros http://nbviewer.ipython.org/gist/Juanlu001/7711865
* Estabilidad longitudinal de un Boeing 747 http://nbviewer.ipython.org/github/AlexS12/Mecanica_Vuelo/blob/master/MVII_MatrizSistema.ipynb

_¿Serás tú el siguiente que publique un notebook usando SymPy? ;)_

Si te ha gustado esta clase:

Tweet

---

####
¡Síguenos en Twitter!

###### Follow @Pybonacci   Follow @Alex__S12   Follow @newlawrence

#####
Curso AeroPython por Juan Luis Cano Rodriguez y Alejandro Sáez Mollejo se distribuye bajo una Licencia Creative Commons Atribución 4.0 Internacional.

#####

---
_Las siguientes celdas contienen configuración del Notebook_

_Para visualizar y utlizar los enlaces a Twitter el notebook debe ejecutarse como [seguro](http://ipython.org/ipython-doc/dev/notebook/security.html)_

    File > Trusted Notebook

```python
%%html
<a href="https://twitter.com/Pybonacci" class="twitter-follow-button" data-show-count="false">Follow @Pybonacci</a>

```

Follow @Pybonacci

```python
# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = '../styles/aeropython.css'
HTML(open(css_file, "r").read())
```

/* This template is inspired in the one used by Lorena Barba
in the numerical-mooc repository: https://github.com/numerical-mooc/numerical-mooc
We thank her work and hope you also enjoy the look of the notobooks with this style */

<link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro|Josefin+Sans:400,700,400italic|Ubuntu+Condensed' rel='stylesheet' type='text/css'>

El estilo se ha aplicado =)


# GMRES

## Algunos Ejemplos

```python
import numpy as np
import scipy.linalg as spla
from scipy.sparse.linalg import gmres, LinearOperator
import matplotlib.pyplot as plt
```

### Error

Para analizar convergencia.

```python
error = lambda x, x_a: np.linalg.norm(x - x_a, ord=2, axis=1) / np.linalg.norm(x)
back_error = lambda x, A, b: np.linalg.norm(b - np.dot(A, x), ord=2) / np.linalg.norm(b)
```

Además, para graficar la convergencia.

```python
def plot(err, b_e):
    Ni_e = np.arange(err.shape[-1])
    Ni_b = np.arange(b_e.shape[-1])
    plt.figure(figsize=(12, 6))
    plt.plot(Ni_e, err, 'bo', label=r'$||\mathbf{x} - \mathbf{x}_a||_2/||\mathbf{x}||_2$')
    plt.plot(Ni_b, b_e, 'rx', label=r'$||\mathbf{b} - A\,\mathbf{x}||_2/||\mathbf{b}||_2$')
    plt.yscale('log')
    plt.xlabel('# Iterations')
    plt.ylabel('Error')
    plt.legend()
    plt.grid(True)
    plt.show()
```

### Implementación $QR$ reducido

El algoritmo de Gram-Schmidt para obtener $\hat{Q}$ y $\hat{R}$.

```python
def QR(A, modified=True):
    m, n = A.shape
    Q = np.zeros((m, n))
    R = np.zeros((n, n))
    # Gram-Schmidt orthogonalization
    for j in range(n):
        y = A[:, j]
        if modified:
            for i in range(j):
                R[i, j] = np.dot(Q[:, i], y)
                y = y - R[i, j] * Q[:, i]
        else:
            for i in range(j):
                R[i, j] = np.dot(Q[:, i], A[:, j])
                y = y - R[i, j] * Q[:, i]
        R[j, j] = np.linalg.norm(y)
        Q[:, j] = y / R[j, j]
    return Q, R
```

### Mínimos Cuadrados

- Utilizando $QR$
    1. Obtener $QR$ reducido, es decir, $A=\hat{Q}\,\hat{R}$
    2. Resolver $\hat{R}\,\hat{\mathbf{x}} = \hat{Q}^*\mathbf{b}$

- Utilizando Ecuaciones Normales
    1. Resolver $A^*A\hat{\mathbf{x}} = A^*\mathbf{b}$

```python
def lstsq(A, b, qr=True):
    if qr:
        Q, R = QR(A) # QR reduced
        x = spla.solve_triangular(R, np.dot(Q.T, b)) # Solve Rx=Q^*b
    else:
        x = np.linalg.solve(np.dot(A.T, A), np.dot(A.T, b)) # Solve A^*Ax=A^*b
    return x
```

### Implementación GMRES

Versión modificada de [1].

```python
def GMRES(A, b, x_0=None, tol=1e-10, max_iter=100, show=False, np_ls=False):
    m = b.shape[0]

    M = np.min([m, max_iter])

    X = np.zeros((M + 1, m))
    if x_0 is not None:
        X[0] = x_0

    r_0 = b - np.dot(A, X[0])
    n_r0 = np.linalg.norm(r_0)

    Q = np.zeros((m, m))
    H = np.zeros((m, m))

    Q[:, 0] = r_0 / n_r0

    for k in range(M):

        # Arnoldi Iteration
        y = np.dot(A, Q[:, k])
        for j in np.arange(k+1):
            H[j, k] = np.dot(Q[:,j], y)
            y = y - H[j, k] * Q[:,j]

        # Build H matrix and fill Q_k
        if k + 1 < m:
            H[k+1, k] = np.linalg.norm(y)

            if np.abs(H[k+1, k]) > 1e-16:
                Q[:, k+1] = y / H[k+1, k]

            e_1 = np.zeros((k + 1) + 1)
            e_1[0] = 1
            H_tilde = H[:(k + 1) + 1, :k + 1]
            s = k + 1
        else: # When we use all Krylov subspace
            H_tilde = H[:k + 1, :k + 1]
            s = m + 1

        # Solving the 'SMALL' least squares problem.
        if np_ls:
            c_k, _, _, _ = np.linalg.lstsq(H_tilde, n_r0 * e_1, rcond=None) # Using Numpy Implementation (faster)
        else:
            c_k = lstsq(H_tilde, n_r0 * e_1) # Using algorithms seen in classes (slower)

        if show:
            print("k:", k)
            print("H:\n", H)
            print("H_t:\n", H_tilde)
            print("e_1:\n", e_1)
            print("c_k:\n", c_k)
            print("Q_k:\n", Q[:,:s])
            print()

        X[k+1] = X[0] + np.dot(Q[:, :s], c_k)

        if np.linalg.norm(np.dot(A, X[k+1]) - b) < tol:
            X = X[:k + 2]
            break

    return X
```

## Ejemplo 1

Matriz $A$ aleatoria.

```python
np.random.seed(1234567890)
m = 50
A_1 = np.random.rand(m, m)
b_1 = np.dot(A_1, np.ones(m))
```

#### Solver de ```numpy```

```python
x_1 = np.linalg.solve(A_1, b_1)
x_1
```

    array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

#### Implementación

```python
x_g_1 = GMRES(A_1, b_1, tol=1e-10)
x_g_1[-1]
```

    array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

#### GMRES ```scipy```

```python
x_s_1, _ = gmres(A_1, b_1)
x_s_1
```

    array([1.19763667, 1.15258581, 0.75789253, 0.98707487, 1.10148399,
           0.92463966, 1.01559199, 1.13767641, 1.00459424, 1.17393247,
           0.90477812, 0.94783382, 0.98404039, 1.15778863, 0.94996579,
           0.98817646, 1.16153322, 0.94501359, 1.0144308 , 1.06213658,
           1.12572779, 0.92815843, 0.95481232, 0.83542311, 1.00947238,
           0.97835678, 0.95642767, 0.93572698, 0.9990528 , 1.08851003,
           0.78950694, 0.94941065, 0.79857037, 1.11264603, 1.13777757,
           0.97964287, 1.00943078, 1.00065634, 0.76325745, 1.04798721,
           1.21140939, 1.07522482, 0.83799066, 0.98403268, 1.08818604,
           1.01637778, 0.97685978, 0.87574449, 0.94158985, 0.9825196 ])

#### Convergencia

```python
b_e_1 = np.array([back_error(x_i, A_1, b_1) for x_i in x_g_1])
err_1 = error(x_1, x_g_1)
```

```python
plot(err_1, b_e_1)
```

¿Podemos Mejorar la convergencia? Intentemos utilizando *GMRES* con reinicio.

```python
p = 20
M = 5
X_r = np.zeros(((M * p), m))
x_0_ = None
for r in range(M):
    x_g_1_a = GMRES(A_1, b_1, x_0=x_0_, max_iter=p, tol=1e-16)
    X_r[r * p: (r + 1) * p] = x_g_1_a[1:]
    x_0_ = x_g_1_a[-1]
```

```python
b_e_1_a = np.array([back_error(x_i, A_1, b_1) for x_i in X_r])
err_1_a = error(x_1, X_r)
```

```python
plot(err_1_a, b_e_1_a)
```

¿Problema? Quizás la matriz $A$ es mal condicionada.

```python
np.linalg.cond(A_1)
```

    1976.6502616108346

## Ejemplo 2

Modifiquemos levemente nuestra matriz $A$

```python
A_2 = A_1 + m * np.eye(m)
b_2 = np.dot(A_2, np.ones(m))
```

#### Solver de ```numpy```

```python
x_2 = np.linalg.solve(A_2, b_2)
x_2
```

    array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

#### Implementación

```python
x_g_2 = GMRES(A_2, b_2, tol=1e-10)
x_g_2[-1]
```

    array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

#### GMRES ```scipy```

```python
x_s_2, _ = gmres(A_2, b_2)
x_s_2
```

    array([1.00000283, 0.99999891, 0.99999816, 1.00000288, 0.99999952,
           1.00000131, 0.99999789, 0.99999908, 1.00000076, 0.9999992 ,
           0.99999774, 0.99999935, 1.00000072, 0.99999843, 0.99999799,
           0.99999967, 1.00000151, 0.99999965, 1.00000197, 0.99999941,
           1.00000173, 1.00000211, 1.00000044, 1.00000144, 0.99999903,
           0.99999915, 0.99999907, 1.0000025 , 1.0000014 , 0.99999941,
           0.99999976, 1.00000038, 0.99999592, 0.99999985, 1.00000254,
           1.00000003, 0.99999703, 0.99999811, 1.00000046, 0.99999981,
           0.99999927, 0.99999954, 0.99999972, 1.00000031, 1.00000321,
           1.00000122, 0.99999899, 1.00000174, 0.99999856, 1.00000003])

```python
b_e_2 = np.array([back_error(x_i, A_2, b_2) for x_i in x_g_2])
err_2 = error(x_2, x_g_2)
```

```python
plot(b_e_2, err_2)
```

## Ejemplo 3: Evitar construir $A$ explícitamente

Supongamos necesitamos resolver el sistema $A\mathbf{x}=\mathbf{b}$, con $A\in\mathbb{R}^{n\times n}$ tridiagonal, y $\mathbf{b}\in\mathbb{R}^n$. ¿Necesitamos calcular el producto matriz vector $A\mathbf{v}$ para $\mathbf{v}\in\mathbb{R}^n$ cualquiera?

La estructura de $A$ tridiagonal es:
\begin{equation}
    A=
    \begin{bmatrix}
        a_{1} & b_{1}\\
        c_{1}& a_{2} & b_{2}\\
        &c_{2}&\ddots &\ddots \\
        & & \ddots & \ddots &b_{n-1}\\
        & & & c_{n-1} & a_{n}
    \end{bmatrix}
\end{equation}

### Construcción $A$ tridiagonal

```python
def buildA(n):
    np.random.seed(1234)
    # Random arrays to fill diagonals
    a_ = np.random.rand(n)
    b_ = np.random.rand(n-1)
    c_ = np.random.rand(n-1)
    # Temporal matrices to add diagonals
    a_tmp = np.zeros((n, n))
    b_tmp = np.zeros_like(a_tmp)
    c_tmp = np.zeros_like(a_tmp)
    # Diagonals
    a_tmp = a_ * np.eye(n) # Main diagonal
    b_tmp[:-1,1:] = b_ * np.eye(n-1) # Above diagonal
    c_tmp[1:,:-1] = c_ * np.eye(n-1) # Below diagonal

    # Build A
    A = a_tmp + b_tmp + c_tmp

    del a_tmp, b_tmp, c_tmp

    return A
```

```python
n = 5
A = buildA(n)
```

```python
# Just show when A is small...
if n < 7:
    print(A)
```

    [[0.19151945 0.27259261 0.         0.         0.        ]
     [0.87593263 0.62210877 0.27646426 0.         0.        ]
     [0.         0.35781727 0.43772774 0.80187218 0.        ]
     [0.         0.         0.50099513 0.78535858 0.95813935]
     [0.         0.         0.         0.68346294 0.77997581]]

### Construcción ```afun(v)```

Con esta función queremos calcular el producto matriz vector $A\mathbf{v}$ sin necesidad de ocupar todos los coeficientes de $A$. Dado que la matriz tiene muchas entradas nulas evitamos aplicar el algoritmo completo de multiplicación matriz-vector.

```python
def buildAfun(A, v):
    # Get matrix diagonals
    a = np.diag(A)
    b = np.diag(A, 1)
    c = np.diag(A, -1)

    # Vector size
    n = v.shape[0]
    u = np.zeros(n) # Output vector

    u[0] = a[0] * v[0] + b[0] * v[1] # First element of u
    u[-1] = c[-1] * v[-2] + a[-1] * v[-1] # Last element of u
    for i in range(1, n-1):
        u[i] = c[i-1] * v[i-1] + a[i] * v[i] + b[i] * v[i+1] # Inside u...
    return u
```

Probemos el tiempo de cómputo multiplicando por un vector aleatorio.

```python
n_test = 10000 # For large n we can see the difference in computational time
A_test = buildA(n_test)
v = np.random.rand(n_test)
afun_test = lambda v: buildAfun(A_test, v)
```

```python
%timeit np.dot(A_test, v)
```

    52 ms ± 1.22 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%timeit afun_test(v)
```

    22.1 ms ± 688 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
Av_1 = np.dot(A_test, v)
Av_2 = afun_test(v)
print(np.linalg.norm(Av_1- Av_2))
```

    4.8266229755959866e-15

Notamos que calcular el producto punto es mucho más costoso que utilizar nuestra función. Luego vemos que el resultado de ambos $A\mathbf{v}$ es similar. Importante que la diferencia de tiempo computacional se aprecia para valores de $n$ grandes, ya que ```numpy``` ya tiene considerablemente optimizado el producto matriz-vector.

### "Matrix-free" GMRES

Modificamos nuestro algoritmo para que no calcule el producto matriz-vector y utilizamos nuestra ```afun(v)```.

```python
def GMRES2(fun, b, x_0=None, tol=1e-10, max_iter=100, show=False, np_ls=False):
    m = b.shape[0]
    M = np.min([m, max_iter])
    X = np.zeros((M + 1, m))
    if x_0 is not None:
        X[0] = x_0

    r_0 = b - fun(X[0])
    n_r0 = np.linalg.norm(r_0)
    Q = np.zeros((m, m))
    H = np.zeros((m, m))
    Q[:, 0] = r_0 / n_r0

    for k in range(M):

        # Arnoldi Iteration
        y = fun(Q[:, k])
        for j in np.arange(k+1):
            H[j, k] = np.dot(Q[:,j], y)
            y = y - H[j, k] * Q[:,j]

        # Build H matrix and fill Q_k
        if k + 1 < m:
            H[k+1, k] = np.linalg.norm(y)

            if np.abs(H[k+1, k]) > 1e-16:
                Q[:, k+1] = y / H[k+1, k]

            e_1 = np.zeros((k + 1) + 1)
            e_1[0] = 1
            H_tilde = H[:(k + 1) + 1, :k + 1]
            s = k + 1
        else: # When we use all Krylov subspace
            H_tilde = H[:k + 1, :k + 1]
            s = m + 1

        # Solving the 'SMALL' least squares problem.
        if np_ls:
            c_k, _, _, _ = np.linalg.lstsq(H_tilde, n_r0 * e_1, rcond=None) # Using Numpy Implementation
        else:
            c_k = lstsq(H_tilde, n_r0 * e_1) # Using algorithms seen in classes

        if show:
            print("k:", k)
            print("H:\n", H)
            print("H_t:\n", H_tilde)
            print("e_1:\n", e_1)
            print("c_k:\n", c_k)
            print("Q_k:\n", Q[:,:s])
            print()

        X[k+1] = X[0] + np.dot(Q[:, :s], c_k)

        if np.linalg.norm(fun(X[k+1]) - b) < tol:
            X = X[:k + 2]
            break

    return X
```

```python
n = 100
A_3 = buildA(n)
b_3 = np.dot(A_3, np.arange(1, n+1))
afun = lambda v: buildAfun(A_3, v)
```

```python
x_3 = GMRES(A_3, b_3, np_ls=True, max_iter=n)
```

```python
x_3[-1]
```

    array([  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,  11.,
            12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,  21.,  22.,
            23.,  24.,  25.,  26.,  27.,  28.,  29.,  30.,  31.,  32.,  33.,
            34.,  35.,  36.,  37.,  38.,  39.,  40.,  41.,  42.,  43.,  44.,
            45.,  46.,  47.,  48.,  49.,  50.,  51.,  52.,  53.,  54.,  55.,
            56.,  57.,  58.,  59.,  60.,  61.,  62.,  63.,  64.,  65.,  66.,
            67.,  68.,  69.,  70.,  71.,  72.,  73.,  74.,  75.,  76.,  77.,
            78.,  79.,  80.,  81.,  82.,  83.,  84.,  85.,  86.,  87.,  88.,
            89.,  90.,  91.,  92.,  93.,  94.,  95.,  96.,  97.,  98.,  99.,
           100.])

```python
x_3 = GMRES2(afun, b_3, np_ls=True, max_iter=n)
```

```python
x_3[-1]
```

    array([  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,  11.,
            12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,  21.,  22.,
            23.,  24.,  25.,  26.,  27.,  28.,  29.,  30.,  31.,  32.,  33.,
            34.,  35.,  36.,  37.,  38.,  39.,  40.,  41.,  42.,  43.,  44.,
            45.,  46.,  47.,  48.,  49.,  50.,  51.,  52.,  53.,  54.,  55.,
            56.,  57.,  58.,  59.,  60.,  61.,  62.,  63.,  64.,  65.,  66.,
            67.,  68.,  69.,  70.,  71.,  72.,  73.,  74.,  75.,  76.,  77.,
            78.,  79.,  80.,  81.,  82.,  83.,  84.,  85.,  86.,  87.,  88.,
            89.,  90.,  91.,  92.,  93.,  94.,  95.,  96.,  97.,  98.,  99.,
           100.])

Notar que en ambos casos obtenemos la solución, pero el impacto de ```afun(v)``` se ve cuando tenemos sistemas de ecuaciones muy grandes.

## Comentarios

Recordar que en cada iteración la complejidad de nuestro algoritmo es la complejidad del producto matriz-vector $O(m^2)$ más la resolución del problema de mínimos cuadrados $O(k^3)$ si utilizamos $QR$ por ejemplo. Se omite la obtención de $h_{j,k}$ ya que es un producto punto, es decir, $O(m)$. De acuerdo a [3] podemos reducir la complejidad del problema de mínimos cuadrados aprovechando la estructura de $\tilde{H}_k$ de manera que la mayor parte del costo computacional se encuentra en resolver $\hat{R}\,\mathbf{x}=\hat{Q}^*\mathbf{b}$, que sería $O(k^2)$.

Ahora, nuestro algoritmo tiene como complejidad $k\left(O(m^2) + O(k^2)\right)$, donde el peor caso es $k=m$, o sea obtenemos una complejidad de $O(m^3)$ similar a $PA=LU$ por ejemplo. El objetivo es resolver el problema cuando $k<m$, es decir, con complejidad $O(k^3)$. Notar que el uso de ```afun(v)``` nos ayuda a evitar el costo de la multiplicación matriz-vector.

---

## Ejercicio 4: Ecuación de Sylvester

La ecuación de Sylvester es una ecuación matricial de la forma:
\begin{equation}
    A\,X + X\,B = C,
\end{equation}

donde $A, B, C \in \mathbb{R}^{n \times n}$ son matrices conocidas y $X\in \mathbb{R}^{n \times n}$ nuestra matriz incógnita.

Utilicemos un método iterativo para resolverlo. Algoritmo obtenido de [4].

### Idea 1
\begin{equation}
    \begin{split}
        A\,X + X\,B & = C \\
        A\,X & = C - X\,B
    \end{split}
\end{equation}

Luego iteramos $A\,X_{i+1} = C - X_i\,B$, con $X_0$ una estimación inicial.

### Idea 2
\begin{equation}
    \begin{split}
        A\,X + X\,B & = C \\
        X\,B & = C - A\,X \\
        (X\,B)^T & = (C - A\,X)^T  \\
        B^TX^T &= (C - A\,X)^T
    \end{split}
\end{equation}

Luego iteramos $B^T\,X^T_{i+1} = (C - A\,X_i)^T$, con $X_0$ una estimación inicial.

```python
np.random.seed(2468)
n = 10
A_4 = np.random.rand(n, n) + n * np.eye(n)
B_4 = np.random.rand(n, n)
C_4 = np.random.rand(n, n)
```

```python
def iterativeSylvester(A, B, C, m, alg=1, tol=1e-10):
    X0 = np.zeros_like(A)
    X1 = np.zeros_like(A)
    if alg == 1: # Algorithm 1
        for i in range(m):
            X1 = np.linalg.solve(A, C - np.dot(X0, B))
            X0 = X1
            if np.linalg.norm(np.dot(A, X1) + np.dot(X1, B) - C) < tol:
                break
        return X1
    elif alg == 2: # Algorithm 2
        for i in range(m):
            X1 = np.linalg.solve(B.T, (C - np.dot(A, X0)).T).T
            X0 = X1
            if np.linalg.norm(np.dot(A, X1) + np.dot(X1, B) - C) < tol:
                break
        return X1
```

### Propuesta 1

```python
X_1 = iterativeSylvester(A_4, B_4, C_4, 70)
```

```python
np.linalg.norm(np.dot(A_4, X_1) + np.dot(X_1, B_4) - C_4)
```

    6.53279812709292e-11

### Propuesta 2

```python
X_2 = iterativeSylvester(A_4, B_4, C_4, 10, 2)
```

```python
np.linalg.norm(np.dot(A_4, X_2) + np.dot(X_2, B_4) - C_4)
```

    2.5849271783041154e+22

### Propuesta 3

Utilizando GMRES. En este enfoque, vectorizamos la matriz $C$ para resolver un problema $A\,\mathbf{x}=\mathbf{b}$ donde $\mathbf{b}=\operatorname{vec}(C)$. Además, nuestro "producto matriz vector", será $AX+XB$.

```python
def AXBX(x, A, B):
    X = np.reshape(x, A.shape) # Get flattened version of x and reshape to get X
    out = np.dot(A, X) + np.dot(X, B) # Out "matrix-vector" multiplication
    return out.flatten()

Ax = lambda x: AXBX(x, A_4, B_4)
```

#### Resolviendo con GMRES implementado

```python
x_u = GMRES2(Ax, C_4.flatten(), np_ls=True, max_iter=n ** 2) # C.flatten() is vec(C)...
X_3_u = np.reshape(x_u[-1], A_4.shape)
```

```python
np.linalg.norm(np.dot(A_4, X_3_u) + np.dot(X_3_u, B_4) - C_4)
```

    2.626608384452874e-11

#### Resolviendo con GMRES de ```scipy```

```python
# This is required by Scipy GMRES... It's just a funcion to compute matrix-vector multiplication
afun2 = LinearOperator((n ** 2, n ** 2), matvec = Ax)
x_s, exitCode = gmres(afun2, C_4.flatten(), tol=1e-10)
X_3_s = np.reshape(x_s, A_4.shape)
```

```python
np.linalg.norm(np.dot(A_4, X_3_s) + np.dot(X_3_s, B_4) - C_4)
```

    1.6638877930818026e-10

### Solución del problema

```python
X = spla.solve_sylvester(A_4, B_4, C_4)
```

```python
print("Propuesta 1: ", np.linalg.norm(X_1 - X))
print("Propuesta 2: ", np.linalg.norm(X_2 - X))
print("Propuesta 3a:", np.linalg.norm(X_3_s - X))
print("Propuesta 3b:", np.linalg.norm(X_3_u - X))
```

    Propuesta 1:  4.689803856203944e-12
    Propuesta 2:  1.690995795460254e+21
    Propuesta 3a: 1.6368741557609354e-11
    Propuesta 3b: 2.5798177172434354e-12

La propuesta 2 no converge.
El objetivo del ejercicio es conectar una iteración de punto fijo con una ecuación matricial.

# Referencias
* [1] https://github.com/tclaudioe/Scientific-Computing/blob/master/SC1/10_GMRes.ipynb
* [2] https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.gmres.html
* [3] Trefethen, L. N., & Bau III, D. (1997). Numerical linear algebra (Vol. 50). SIAM.
* [4] https://github.com/tclaudioe/Scientific-Computing/blob/master/SC1/Bonus%20-%20Sylvester%20Equation%20with%20GMRes%20.ipynb