# Latent Variable Models and Variational Bayes

### Preliminaries

- Goal
  - Introduction to latent variable models and variational inference by Free energy minimization
- Materials
  - Mandatory
    - These lecture notes
    - Ariel Caticha (2010), [Entropic Inference](https://arxiv.org/abs/1011.0723)
      - tutorial on entropic inference, which is a generalization to Bayes rule and provides a foundation for variational inference.
  - Optional
    - Bishop (2016), pp. 461-486 (sections 10.1, 10.2 and 10.3)
  - references
    - Blei et al. (2017), [Variational Inference: A Review for Statisticians](https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1285773)
    - Lanczos (1961), [The variational principles of mechanics](https://www.amazon.com/Variational-Principles-Mechanics-Dover-Physics/dp/0486650677)
    - Zhang et al. (2017), [Unifying Message Passing Algorithms Under the Framework of Constrained Bethe Free Energy Minimization](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Zhang-2017-Unifying-Message-Passing-Algorithms.pdf)
    - Dauwels (2007), [On variational message passing on factor graphs](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Dauwels-2007-on-variational-message-passing-on-factor-graphs)
    - Caticha (2010), [Entropic Inference](https://arxiv.org/abs/1011.0723)
    - Shore and Johnson (1980), [Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross-Entropy](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/ShoreJohnson-1980-Axiomatic-Derivation-of-the-Principle-of-Maximum-Entropy.pdf)

### Illustrative Example : Density Modeling for the Old Faithful Data Set

- You're now asked to build a density model for a data set ([Old Faithful](https://en.wikipedia.org/wiki/Old_Faithful), Bishop pg. 681) that clearly is not distributed as a single Gaussian:

###  Unobserved Classes

Consider again a set of observed data $D=\{x_1,\dotsc,x_N\}$

- This time we suspect that there are _unobserved_ class labels that would help explain (or predict) the data, e.g.,
  - the observed data are the color of living things; the unobserved classes are animals and plants.
  - observed are wheel sizes; unobserved categories are trucks and personal cars.
  - observed is an audio signal; unobserved classes include speech, music, traffic noise, etc.

- Classification problems with unobserved classes are called **Clustering** problems. The learning algorithm needs to **discover the classes from the observed data**.

### The Gaussian Mixture Model

- The spread of the data in the illustrative example looks like it could be modeled by two Gaussians. Let's develop a model for this data set.

- Let $D=\{x_n\}$ be a set of observations. We associate a one-hot coded hidden class label $z_n$ with each observation:

$$\begin{equation*}
z_{nk} = \begin{cases} 1 & \text{if } x_n \in \mathcal{C}_k \text{ (the k-th class)}\\
                       0 & \text{otherwise} \end{cases}
\end{equation*}$$

- We consider the same model as we did in the [generative classification lesson](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/Generative-Classification.ipynb#GDA):
$$\begin{align*}
p(x_n | z_{nk}=1) &= \mathcal{N}\left( x_n | \mu_k, \Sigma_k\right)\\
p(z_{nk}=1) &= \pi_k
\end{align*}$$
which can be summarized with the selection variables $z_{nk}$ as
$$\begin{align*}
p(x_n,z_n) &= p(x_n | z_n) p(z_n) = \prod_{k=1}^K (\underbrace{\pi_k \cdot \mathcal{N}\left( x_n | \mu_k, \Sigma_k\right) }_{p(x_n,z_{nk}=1)})^{z_{nk}}
\end{align*}$$

- *Again*, this is the same model  as we defined for the generative classification model: A Gaussian-Categorical model but now with unobserved classes).

- This model (with **unobserved class labels**) is known as a **Gaussian Mixture Model** (GMM).

### The Marginal Distribution for the GMM

- In the literature, the GMM is often introduced the marginal distribution for an _observed_ data point $x_n$, given by

$$\begin{align*}{}
p(x_n) &= \sum_{z_n} p(x_n,z_n)  \\
  &= \sum_{k=1}^K \pi_k \cdot \mathcal{N}\left( x_n | \mu_k, \Sigma_k \right) \tag{B-9.12}
\end{align*}$$

- Full proof as an [exercise](https://nbviewer.org/github/bertdv/BMLIP/blob/master/lessons/exercises/Exercises-Latent-Variable-Models-and-VB.ipynb).

- Eq. B-9.12 reveals the link to the name Gaussian *mixture model*. The priors $\pi_k$ are also called **mixture coefficients**.

###  GMM is a very flexible model

- GMMs are very popular models. They have decent computational properties and are **universal approximators of densities** (as long as there are enough Gaussians of course)

- (In the above figure, the Gaussian components are shown in red and the pdf of the mixture models in blue).

### Latent Variable Models

- The GMM contains both _observed_ variables $\{x_n\}$, (unobserved) _parameters_ $\theta= \{\pi_k,\mu_k, \Sigma_k\}$ _and_ unobserved (synonym: latent, hidden) variables $\{z_{nk}\}$.

- From a Bayesian viewpoint, both latent variables $\{z_{nk}\}$ and parameters $\theta$ are just unobserved variables for which we can set a prior and compute a posterior by Bayes rule.

- Note that $z_{nk}$ has a subscript $n$, hence its value depends not only on the cluster ($k$) but also on the $n$-th observation (in constrast to parameters). These observation-dependent variables are generally a useful tool to encode structure in the model. Here (in the GMM), the latent variables $\{z_{nk}\}$ encode (unobserved) class membership.

- By adding model structure through (equations among) latent variables, we can build very complex models. Unfortunately, inference in latent variable models can also be much more complex than for fully observed models.

### Inference for GMM is Difficult

- We recall here the log-likelihood for the Gaussian-Categorial Model, see [generative classification lesson)](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/Generative-Classification.ipynb)

$$
\log\, p(D|\theta) =  \sum_{n,k} y_{nk} \underbrace{ \log\mathcal{N}(x_n|\mu_k,\Sigma) }_{ \text{Gaussian} } + \underbrace{ \sum_{n,k} y_{nk} \log \pi_k }_{ \text{multinomial} } \,.
$$

- Since the class labels $y_{nk} \in \{0,1\}$ were given, this expression decomposed into a set of simple updates for the Gaussian and multinomial distributions. For both distributions, we have conjugate priors, so inference is easily solved.

- However, for the Gaussian mixture model (same log-likelihood function with $z_{nk}$ replacing $y_{nk}$), the class labels $\{z_{nk}\}$ are _unobserved_ and need to estimated alongside with the parameters.

- There is no conjugate prior on the latent variables for the GMM likelihood function $p(\underbrace{D}_{\{x_n\}}\,|\,\underbrace{\{z_{nk}\},\{\mu_k,\Sigma_k,\pi_k\}}_{\text{all latent variables}})$.

- In this lesson, we introduce an approximate Bayesian inference method known as **Variational Bayes** (VB) (also known as **Variational Inference**) that can be used for Bayesian inference in models with latent variables. Later in this lesson, we will use VB to do inference in the GMM.

- As a motivation for variational inference, we first discuss inference by the **Method of Maximum Relative Entropy**, [(Caticha, 2010)](https://arxiv.org/abs/1011.0723).

### Inference When Information is in the Form of Constraints

- In the [probability theory lesson](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/Probability-Theory-Review.ipynb#Bayes-rule), we recognized Bayes rule as the fundamental rule for learning from data.

- We will now generalize this notion and consider learning as a process that updates a prior into a posterior distribution whenever new information becomes available.

- In this context, *new information* is not necessarily a new observation, but could (for instance) also relate to *constraints* on the posterior distribution.

- For example, consider a model $\prod_n p(x_n,z) = p(z) \prod_n p(x_n|z)$.

- Our prior beliefs about $z$ are represented by $p(z)$. In the following, we will write $q(z)$ to denote a posterior distribution for $z$.

-  We might be interested in obtaining rational posterior beliefs $q(z)$ in consideration of the following additional constraints:
  1. *Data Constraints*: e.g., two observed data points $x_1=5$ and $x_2=3$, which lead to constraints $q(x_1) = \delta(x_1-5)$ and $q(x_2)=\delta(x_2-3)$.
  2. *Form Constraints*: e.g., we only consider the Gamma distribution for $q(z) = \mathrm{Gam}(z|\alpha,\beta)$.
  3. *Factorization Constraints*:, e.g., we consider independent marginals for the posterior distribution: $q(z) = \prod_k q(z_k)$.
  3. *Moment Constraints*: e.g., the first moment of the posterior is given by $\int z q(z) \mathrm{d}z = 3$.

- Note that this is not "just" applying Bayes rule to compute a posterior, which can only deal with data constraints as specified by the likelihood function.

- Note also that observations _can_ be rephrased as constraints on the posterior, e.g., observation $x_1=5$ can be phrased as a constraint $q(x_1)=\delta(x_1-5)$.

- $\Rightarrow$ Updating a prior to a posterior on the basis of constraints on the posterior is more general than updating based on observations alone.

### The Method of Maximum Relative Entropy

- [Caticha (2010)](https://arxiv.org/abs/1011.0723) (based on earlier work by [Shore and Johnson (1980)](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/ShoreJohnson-1980-Axiomatic-Derivation-of-the-Principle-of-Maximum-Entropy.pdf)) developed the **method of maximum (relative) entropy** for rational updating of priors to posteriors when faced with new information in the form of constraints.

- Consider prior beliefs $p(z)$ about $z$. New information in the form of constraints is obtained and we are interested in the "best update" to a posterior $q(z)$.

- In order to define what "best update" means, we assume a ranking function $S[q,p]$ that generates a preference score for each candidate posterior $q$ for a given $p$. The best update from $p$ to $q$ is identified as $q^* = \arg\max_q S[q,p]$.

- Similarly to [Cox' method to deriving probability theory](https://en.wikipedia.org/wiki/Cox%27s_theorem), Caticha introduced the following mild criteria based on a rational principle (the **principle of minimal updating**, see [Caticha 2010](https://arxiv.org/abs/1011.0723)) that the ranking function needs to adhere to:
  1. *Locality*: local information has local effects.
  2. *Coordinate invariance*: the system of coordinates carries no information.
  3. *Independence*: When systems are known to be independent, it should not matter whether they are treated separately or jointly.

- These three criteria **uniquely identify the Relative Entropy** (RE) as the proper ranking function:
$$\begin{align*}
S[q,p] = - \sum_z q(z) \log \frac{q(z)}{p(z)}
\end{align*}$$

- $\Rightarrow$ When information is supplied in the form of constaints on the posterior, we *should* select the posterior that maximizes the Relative Entropy, subject to the constraints. This is the **Method of Maximum (Relative) Entropy** (MRE).

### Relative Entropy, KL Divergence and Free Energy

- Note that the Relative Entropy is technically a *functional*, i.e., a function of function(s) (since it is a function of probability distributions). The calculus of functionals is also called **variational calculus**.
  - See  [Lanczos, The variational principles of mechanics (1961)](https://www.amazon.com/Variational-Principles-Mechanics-Dover-Physics/dp/0486650677) for a great book on variational calculus. For a summary, see Bishop (2016), app.D.
  - It is customary to use square brackets (like $S[q,p]$) for functionals rather than round brackets, which we use for functions.

- Also note the complimentary relation between the Maximum Relative Entropy method and Probability Theory:
  - PT describes how to _represent_ beliefs about events and how to _calculate_ beliefs about joint and conditional events.
  - In contrast, the MRE method describes how to _update_ beliefs when new information becomes available.

- PT and the MRE method are both needed to describe the theory of optimal information processing. (As we will see below, Bayes rule is a special case of the MRE method.)

- In principle, entropies can always be considered as a *relative* score against a reference distribution. For instance, the score $-\sum_{z_i} q(z_i) \log q(z_i)$ can be interpreted as a score against the uniform distribution, i.e., $-\sum_{z_i} q(z_i) \log q(z_i) \propto -\sum_{z_i} q(z_i) \log \frac{q(z_i)}{\mathrm{Uniform(z_i)}}$. Therefore, the "method of maximum relative entropy" is often simply known as the "method of maximum entropy".

- The negative relative entropy is known as the **Kullback-Leibler** (KL) divergence:
$$
\mathrm{KL}[q,p] \triangleq \sum_z q(z) \log \frac{q(z)}{p(z)} \tag{B-1.113}
$$

- The [Gibbs inequality](https://en.wikipedia.org/wiki/Gibbs%27_inequality) ([proof](https://en.wikipedia.org/wiki/Gibbs%27_inequality#Proof)), is a famous theorem in information theory that states that
$$\mathrm{KL}[q,p] \geq 0 $$
with equality only iff $p=q$.

  - The KL divergence can be interpreted as a "distance" between two probability distributions. Note however that the KL divergence is an asymmetric distance measure, i.e., in general $\mathrm{KL}[q,p] \neq \mathrm{KL}[p,q]$.

- We also introduce here the notion of a (variational) **Free Energy** (FE) functional, which is just a generalization of the KL-divergence that allows the prior to be unnormalized. Let $f(z) = Z \cdot p(z)$ be an unnormalized distribution, then the FE is defined as
$$\begin{align*}
F[q,p] &\triangleq \sum_z q(z) \log \frac{q(z)}{f(z)} \\
&= \sum_z q(z) \log \frac{q(z)}{Z\cdot p(z)} \\
&= \underbrace{\sum_z q(z) \log \frac{q(z)}{p(z)}}_{\text{KL divergence }\geq 0} - \log Z
\end{align*}$$

- Since the second term ($\log Z$) is constant, FE minimization (subject to constraints) with respect to $q$ leads to the same posteriors as KL minimization and RE maximization.

- If we are only interested in minimizing FE with respect to $q$, then we'll write $F[q]$ (rather than $F[q,p]$) fo brevity.

- In this class, we prefer to discuss inference in terms of minimizing Free Energy (subject to the constraints) rather than maximizing Relative Entropy, but note that these two concepts are equivalent.

### Example KL divergence for Gaussians

source: By Mundhenk at English Wikipedia, CC BY-SA 3.0, Link

### The Free Energy Functional and Variational Bayes

- Let's get back to the issue of doing inference for models with latent variables (such as the GMM).

- Consider a generative model specified by $p(x,z)$ where $x$ and $z$ represent the observed and unobserved variables, respectively.

- Assume further that $x$ has been observed as $x=\hat{x}$ and we are interested in performing inference, i.e., we want to compute the posterior $p(z|x=\hat{x})$ for the latent variables and we want to compute the evidence $p(x=\hat{x})$.

- According to the Method of Maximum Relative Entropy, in order to find the correct posterior $q(x,z)$, we should minimize

$$
\mathrm{F}[q] = \sum_{x,z} q(x,z) \log \frac{q(x,z)}{p(x,z)}\,, \quad \text{subject to data constraint }x=\hat{x}
$$

- The data constraint $x=\hat{x}$ fixes posterior $q(x) = \delta(x-\hat{x})$, so the Free Energy evaluates to

$$\begin{align*}
F[q] &= \sum_{z} \sum_{x}q(z|x)q(x) \log \frac{q(z|x) q(x)}{p(z|x) p(x)} \\
  &= \sum_{z} \sum_{x} q(z|x)\delta(x-\hat{x}) \log \frac{q(z|x)\delta(x-\hat{x})}{p(z|x) p(x)} \\
  &= \sum_{z} q(z|\hat{x}) \log \frac{q(z|\hat{x})}{p(z|\hat{x}) p(\hat{x}) } \\
  &=  \underbrace{\sum_{z}q(z|\hat{x}) \log \frac{q(z|\hat{x})}{p(z|\hat{x})}}_{\text{KL divergence }\geq 0} - \underbrace{\log p(\hat{x})  }\tag{B-10.2
}_{\text{log-evidence}}\end{align*}$$

- The log-evidence term does not depend on $q$. Hence, the global minimum $q(z|\hat{x})^* \triangleq \arg\min_q F[q]$ is obtained for $$q^*(z|\hat{x}) = p(z|\hat{x})\,,$$ which is the correct **Bayesian posterior**.

- Furthermore, the minimal free energy $$F^* \triangleq F[q^*] = -\log p(\hat{x})$$ equals minus **log-evidence**. (Or, equivalently, the evidence is given by $p(\hat{x}) = \exp\left(-F[q^*] \right)$.)

$\Rightarrow$ Bayesian inference can be executed by FE minimization.

-  This is an amazing result!  Note that FE minimization transforms an inference problem (that involves integration) to an optimization problem! Generally, optimization problems are easier to solve than integration problems.

- Executing inference by minimizing the variational FE functional is called **Variational Bayes** (VB) or variational inference.

-  (As an aside), note that Bishop introduces in Eq. B-10.2 an _Evidence Lower BOund_ (in modern machine learning literature abbreviated as **ELBO**) $\mathcal{L}(z)$ that equals the _negative_ FE. We prefer to discuss variational inference in terms of a free energy (but it is the same story as he discusses with the ELBO).

###  FE Minimization as Approximate Bayesian Inference

- In the rest of this lesson, we are concerned with how to execute FE minimization (FEM) w.r.t $q$ for the functional
$$
 F[q] = \sum_{z}q(z) \log \frac{q(z)}{p(z|x)} - \log p(x)
 $$
where $x$ has been observed and $z$ represent all latent variables.
  - To keep the notation uncluttered, in the following we write $x$ rather than $\hat{x}$, and we write simply $q(z)$ (rather than $q(z|\hat{x})$) for the posterior.

- Due to restrictions in our optimization algorithm, we are often unable to fully minimize the FE to the global minimum $q^*(z)$, but rather get stuck in a local minimum $\hat{q}(z)$.

- Note that, since $\mathrm{KL}[q(z),p(z|x)]\geq 0$ for any $q(z)$, the FE is always an upperbound on (minus) log-evidence, i.e.,
$$
F[q] \geq -\log p(x) \,.
$$

- In practice, even if we cannot attain the global minimum, we can still use the local minimum $\hat{q}(z) \approx \arg\min_q F[q]$ to accomplish **approximate Bayesian inference**:

$$\begin{align*}
   \text{posterior: } \hat{q}(z) &\approx p(z|x) \\
   \text{evidence: } p(x) &\approx \exp\left( -F[\hat{q}]\right)
    \end{align*}$$

### Constrained FE Minimization

- Generally speaking, it can be very helpful in the FE minimization task to add some additional constraints on $q(z)$ (i.e., in addition to the data constraints).

- Once we add constraints to $q$ (in addition to data constraints), we are no longer guaranteed that the minimum of the (constrained) FE coincides with the solution by Bayes rule (which only takes data as constraints). So again, at best constrained FEM is only an **approximation to Bayes rule**.

- There are three important cases of adding constraints to $q(z)$ that often alleviates the FE minimization task:

#### 1. mean-field factorization
- We constrain the posterior to factorize into a set of _independent_ factors, i.e.,
$$
q(z) = \prod_{j=1}^m q_j(z_j)\,, \tag{B-10.5}
$$

####  2. fixed-form parameterization
- We constrain the posterior to be part of a parameterized probability distribution, e.g., $$q(z) = \mathcal{N}\left( z | \mu, \Sigma \right)\,.$$
  - In this case, the functional minimization problem for $F[q]$ reduces to the minimization of a _function_ $F(\mu,\Sigma)$ w.r.t. its parameters.

#### 3. the Expectation-Maximization (EM) algorithm
- We place some constraints both on the prior and posterior for $z$ ([to be discussed in the Optional Slides](#EM-Algorithm)) that simplifies FE minimization to maximum-likelihood estimation.

### FE Minimization with Mean-field Factorization Constraints: the CAVI Approach

- Let's work out FE minimization with additional mean-field constraints (=full factorization) constraints:  $$q(z) = \prod_{j=1}^m q_j(z_j)\,.$$
  - In other words, the posteriors for $z_j$ are all considered independent. This is a strong constraint but leads often to good solutions.

- Given the mean-field constraints, it is possible to derive the following expression for the optimal solutions $q_j^*(z_j)$, for $j=1,\ldots,m$:

\begin{equation*} \tag{B-10.9}
\boxed{
\begin{aligned}
\log q_j^*(z_j) &\propto \mathrm{E}_{q_{-j}^*}\left[ \log p(x,z) \right]  \\
  &= \underbrace{\sum_{z_{-j}} q_{-j}^*(z_{-j}) \underbrace{\log p(x,z)}_{\text{"field"}}}_{\text{"mean field"}}
\end{aligned}}
\end{equation*}

where we defined $q_{-j}^*(z_{-j}) \triangleq q_1^*(z_1)q_2^*(z_2)\cdots q_{j-1}^*(z_{j-1})q_{j+1}^*(z_{j+1})\cdots q_m^*(z_m)$.

- **Proof** (from [Blei, 2017](https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1285773)): We first rewrite the FE as a function of $q_j(z_j)$ only:
  $$ F[q_j] = \mathbb{E}_{q_{j}}\left[ \mathbb{E}_{q_{-j}}\left[ \log p(x,z_j,z_{-j})\right]\right] - \mathbb{E}_{q_j}\left[ \log q_j(z_j)\right] + \mathtt{const.}\,,$$
  where the constant holds all terms that do not depend on $z_j$. This expression can be written as
  $$ F[q_j] = \sum_{z_j} q_j(z_j) \log \frac{q_j(z_j)}{\exp\left( \mathbb{E}_{q_{-j}}\left[ \log p(x,z_j,z_{-j})\right]\right)}$$
  which is a KL-divergence that is minimized by Eq. B-10.9.  (end proof)

- This is not yet a full solution to the FE minimization task since the solution $q_j^*(z_j)$ depends on expectations that involve other solutions $q_{i\neq j}^*(z_{i \neq j})$, and each of these other solutions $q_{i\neq j}^*(z_{i \neq j})$ depends on an expection that involves $q_j^*(z_j)$.

- In practice, we solve this chicken-and-egg problem by an iterative approach: we first initialize all $q_j(z_j)$ (for $j=1,\ldots,m$) to an appropriate initial distribution and then cycle through the factors in turn by solving eq.B-10.9 and update $q_{-j}^*(z_{-j})$ with the latest estimates. (See [Blei, 2017](https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1285773), Algorithm 1, p864).

- This algorithm for approximating Bayesian inference is known **Coordinate Ascent Variational Inference** (CAVI).

### Example: FEM for the Gaussian Mixture Model (CAVI Approach)

- Let's get back to the illustrative example at the beginning of this lesson: we want to do [density modeling for the Old Faithful data set](#illustrative-example).

#### model specification

- We consider a Gaussian Mixture Model, specified by
$$\begin{align*}
p(x,z|\theta) &= p(x|z,\mu,\Lambda)p(z|\pi) \\
  &= \prod_{n=1}^N \prod_{k=1}^K \left(\pi_k \cdot \mathcal{N}\left( x_n | \mu_k, \Lambda_k^{-1}\right)\right)^{z_{nk}} \tag{B-10.37,38}
\end{align*}$$
with tuning parameters $\theta=\{\pi_k, \mu_k,\Lambda_k\}$.

- Let us introduce some priors for the parameters. We factorize the prior and choose conjugate distributions by
$$
p(\theta) = p(\pi,\mu,\Lambda) = p(\pi) p(\mu|\Lambda) p(\Lambda)
$$
with
$$\begin{align*}
p(\pi) &= \mathrm{Dir}(\pi|\alpha_0) = C(\alpha_0) \prod_k \pi_k^{\alpha_0-1} \tag{B-10.39}\\
p(\mu|\Lambda) &= \prod_k \mathcal{N}\left(\mu_k | m_0, \left( \beta_0 \Lambda_k\right)^{-1} \right) \tag{B-10.40}\\
p(\Lambda) &= \prod_k \mathcal{W}\left( \Lambda_k | W_0, \nu_0 \right) \tag{B-10.40}
\end{align*}$$
where $\mathcal{W}\left( \cdot \right)$ is a [Wishart distribution](https://en.wikipedia.org/wiki/Wishart_distribution) (i.e., a multi-dimensional Gamma distribution).

- The full generative model is now specified by
$$
p(x,z,\pi,\mu,\Lambda) = p(x|z,\mu,\Lambda) p(z|\pi) p(\pi) p(\mu|\Lambda) p(\Lambda) \tag{B-10.41}
$$
with hyperparameters $\{ \alpha_0, m_0, \beta_0, W_0, \nu_0\}$.

#### inference

- Assume that we have observed $D = \left\{x_1, x_2, \ldots, x_N\right\}$ and are interested to infer the posterior distribution for the tuning parameters:
$$
p(\theta|D) = p(\pi,\mu,\Lambda|D)
$$

- The (perfect) Bayesian solution is
$$
p(\theta|D) = \frac{p(x=D,\theta)}{p(x=D)} = \frac{\sum_z p(x=D,z,\pi,\mu,\Lambda)}{\sum_z \sum_{\pi} \iint p(x=D,z,\pi,\mu,\Lambda) \,\mathrm{d}\mu\mathrm{d}\Lambda}
$$
but this is intractable (See [Blei (2017), p861, eqs. 8 and 9](https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1285773)).

- Alternatively, we can use **FE minimization with factorization constraint**
$$\begin{equation}
q(\theta) = q(z) \cdot q(\pi,\mu,\Lambda) \tag{B-10.42}
\end{equation}$$
on the posterior. For the specified model, this leads to FE minimization wrt the hyperparameters, i.e., we need to minimize the function $F(\alpha_0, m_0, \beta_0, W_0, \nu_0)$.

- Bishop shows that the equations for the [optimal solutions (Eq. B-10.9)](#optimal-solutions) are analytically solvable for the GMM as specified above, leading to the following variational update equations (for $k=1,\ldots,K$):
$$
\begin{align*}
\alpha_k &= \alpha_0 + N_k  \tag{B-10.58} \\
\beta_k &= \beta_0 + N_k  \tag{B-10.60} \\
m_k &= \frac{1}{\beta_k} \left( \beta_0 m_0 + N_k \bar{x}_k \right) \tag{B-10.61} \\
W_k^{-1} &= W_0^{-1} + N_k S_k + \frac{\beta_0 N_k}{\beta_0 + N_k}\left( \bar{x}_k - m_0\right) \left( \bar{x}_k - m_0\right)^T \tag{B-10.62} \\
\nu_k &= \nu_0 + N_k \tag{B-10.63}
\end{align*}
$$
where we used
$$
\begin{align*}
\log \rho_{nk} &= \mathbb{E}\left[ \log \pi_k\right] + \frac{1}{2}\mathbb{E}\left[ \log | \Lambda_k | \right] - \frac{D}{2} \log(2\pi) \\
 & \qquad - \frac{1}{2}\mathbb{E}\left[(x_k - \mu_k)^T \Lambda_k(x_k - \mu_k)  \right]  \tag{B-10.46} \\
r_{nk} &= \frac{\rho_{nk}}{\sum_{j=1}^K \rho_{nj}} \tag{B-10.49} \\
N_k &= \sum_{n=1}^N r_{nk} x_n \tag{B-10.51} \\
\bar{x}_k &= \frac{1}{N_k} \sum_{n=1}^N r_{nk} x_n \tag{B-10.52} \\
S_k &= \frac{1}{N_k} \sum_{n=1}^N r_{nk} \left( x_n - \bar{x}_k\right) \left( x_n - \bar{x}_k\right)^T \tag{B-10.53}
\end{align*}
$$

- Exam guide: Working out FE minimization for the GMM to these update equations (eqs B-10.58 through B-10.63) is not something that you need to reproduce without assistance at the exam. Rather, the essence is that *it is possible* to arrive at closed-form variational update equations for the GMM. You should understand though how FEM works conceptually and in principle be able to derive variational update equations for very simple models that do not involve clever mathematical tricks.

### Code Example: FEM for GMM on Old Faithfull data set

- Below we exemplify training of a Gaussian Mixture Model on the Old Faithful data set by Free Energy Minimization, using the constraints as specified above, e.g., [(B-10.42)](#mf-constraint).

```julia
using Pkg;Pkg.activate("probprog/workspace");Pkg.instantiate()
IJulia.clear_output();
```

```julia
using DataFrames, CSV, LinearAlgebra, PDMats, SpecialFunctions
include("scripts/gmm_plot.jl") # Holds plotting function
old_faithful = CSV.read("datasets/old_faithful.csv",DataFrame);
X = convert(Matrix{Float64}, [old_faithful[!,1] old_faithful[!,2]]');#data matrix
N = size(X, 2) #number of observations
K = 6

function sufficientStatistics(X,r,k::Int) #function to compute sufficient statistics
    N_k = sum(r[k,:])
    hat_x_k = sum([r[k,n]*X[:,n] for n in 1:N]) ./ N_k
    S_k = sum([r[k,n]*(X[:,n]-hat_x_k)*(X[:,n]-hat_x_k)' for n in 1:N]) ./ N_k
    return N_k, hat_x_k, S_k
end

function updateMeanPrecisionPi(m_0,β_0,W_0,ν_0,α_0,r) #variational maximisation function
    m = Array{Float64}(undef,2,K) #mean of the clusters
    β = Array{Float64}(undef,K) #precision scaling for Gausian distribution
    W = Array{Float64}(undef,2,2,K) #precision prior for Wishart distributions
    ν = Array{Float64}(undef,K) #degrees of freedom parameter for Wishart distribution
    α = Array{Float64}(undef,K) #Dirichlet distribution parameter
    for k=1:K
        sst = sufficientStatistics(X,r,k)
        α[k] = α_0[k] + sst[1]; β[k] = β_0[k] + sst[1]; ν[k] = ν_0[k] .+ sst[1]
        m[:,k] = (1/β[k])*(β_0[k].*m_0[:,k] + sst[1].*sst[2])
        W[:,:,k] = inv(inv(W_0[:,:,k])+sst[3]*sst[1] + ((β_0[k]*sst[1])/(β_0[k]+sst[1])).*(sst[2]-m_0[:,k])*(sst[2]-m_0[:,k])')
    end
    return m,β,W,ν,α
end

function updateR(Λ,m,α,ν,β) #variational expectation function
    r = Array{Float64}(undef,K,N) #responsibilities
    hat_π = Array{Float64}(undef,K)
    hat_Λ = Array{Float64}(undef,K)
    for k=1:K
        hat_Λ[k] = 1/2*(2*log(2)+logdet(Λ[:,:,k])+digamma(ν[k]/2)+digamma((ν[k]-1)/2))
        hat_π[k] = exp(digamma(α[k])-digamma(sum(α)))
        for n=1:N
           r[k,n] = hat_π[k]*exp(-hat_Λ[k]-1/β[k] - (ν[k]/2)*(X[:,n]-m[:,k])'*Λ[:,:,k]*(X[:,n]-m[:,k]))
        end
    end
    for n=1:N
        r[:,n] = r[:,n]./ sum(r[:,n]) #normalize to ensure r represents probabilities
    end
    return r
end

max_iter = 15
#store the inference results in these vectors
ν = fill!(Array{Float64}(undef,K,max_iter),3)
β = fill!(Array{Float64}(undef,K,max_iter),1.0)
α = fill!(Array{Float64}(undef,K,max_iter),0.01)
R = Array{Float64}(undef,K,N,max_iter)
M = Array{Float64}(undef,2,K,max_iter)
Λ = Array{Float64}(undef,2,2,K,max_iter)
clusters_vb = Array{Distribution}(undef,K,max_iter) #clusters to be plotted
#initialize prior distribution parameters
M[:,:,1] = [[3.0; 60.0];[4.0; 70.0];[2.0; 50.0];[2.0; 60.0];[3.0; 100.0];[4.0; 80.0]]
for k=1:K
    Λ[:,:,k,1] = [1.0 0;0 0.01]
    R[k,:,1] = 1/(K)*ones(N)
    clusters_vb[k,1] = MvNormal(M[:,k,1],PDMats.PDMat(convert(Matrix,Hermitian(inv(ν[1,1].*Λ[:,:,k,1])))))
end
#variational inference
for i=1:max_iter-1
    #variational expectation
    R[:,:,i+1] = updateR(Λ[:,:,:,i],M[:,:,i],α[:,i],ν[:,i],β[:,i])
    #variational minimisation
    M[:,:,i+1],β[:,i+1],Λ[:,:,:,i+1],ν[:,i+1],α[:,i+1] = updateMeanPrecisionPi(M[:,:,i],β[:,i],Λ[:,:,:,i],ν[:,i],α[:,i],R[:,:,i+1])
    for k=1:K
        clusters_vb[k,i+1] = MvNormal(M[:,k,i+1],PDMats.PDMat(convert(Matrix,Hermitian(inv(ν[k,i+1].*Λ[:,:,k,i+1])))))
    end
end

subplot(2,3,1); plotGMM(X, clusters_vb[:,1], R[:,:,1]); title("Initial situation")
for i=2:6
    subplot(2,3,i)
    plotGMM(X, clusters_vb[:,i*2], R[:,:,i*2]); title("After $(i*2) iterations")
end
PyPlot.tight_layout();
```

The generated figure looks much like Figure 10.6 in Bishop. The plots show results for Variational Bayesian
mixture of K = 6 Gaussians applied to the Old Faithful data set, in
which the ellipses denote the one
standard-deviation density contours
for each of the components, and the
color coding of the data points reflects the "soft" class label assignments. Components whose expected
mixing coefficient are numerically indistinguishable from zero are not
plotted. Note that this procedure learns the number of classes (two), learns the class label for each observation, and learns the mean and variance for the Gaussian data distributions.

###  Free Energy Decompositions and Fixed-form Constraints

-  Making use of $p(x,z) = p(z|x)p(x) = p(x|z)p(z)$, the FE functional can be rewritten as

$$\begin{align*}
\mathrm{F}[q] &= \underbrace{-\sum_z q(z) \log p(x,z)}_{\text{energy}} - \underbrace{\sum_z q(z) \log \frac{1}{q(z)}}_{\text{entropy}} \tag{EE} \\
&= \underbrace{\sum_z q(z) \log \frac{q(z)}{p(z|x)}}_{\text{KL divergence}\geq 0} - \underbrace{\log p(x)}_{\text{log-evidence}} \tag{DE}\\
&= \underbrace{\sum_z q(z)\log\frac{q(z)}{p(z)}}_{\text{complexity}} - \underbrace{\sum_z q(z) \log p(x|z)}_{\text{accuracy}}  \tag{CA}
\end{align*}$$

- These decompositions are very insightful (we will revisit them later) and we will label them respectively as _energy-entropy_ (EE), _divergence-evidence_ (DE), and _complexity-accuracy_ (CA) decompositions.

- In the [Bayesian Machine Learning](https://nbviewer.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/Bayesian-Machine-Learning.ipynb) lecture, we discussed the CA decomposition of Bayesian model evidence to support the interpretation of evidence as a model performance criterion. Here, we recognize that FE allows a similar CA decomposition: minimizing FE increases data accuracy and decreases model complexity.

- The CA decomposition makes use of the prior $p(z)$ and likelihood $p(x|z)$, both of which are selected by the engineer, so the FE can be evaluated with this decomposition!

- If we now assume some parametric form for $q(z)$, e.g. $q(z) = \mathcal{N}(z\mid \mu, \Sigma)$, then the FE functional degenerates to a **regular function** $F(\mu,\Sigma)$. In principle, this function can be evaluated by the CA decomposition and minimized by standard (function) minimization methods.

- The EE decomposition provides a link to a second law of thermodynamics: Minimizing FE leads to entropy maximization.

###  Summary

- Latent variable models (LVM) contain a set of unobserved variables whose size grows with the number of observations.

- LVMs can model more complex phenomena than fully observed models, but inference in LVM models is usually not analytically solvable.

- The Free Energy (FE) functional transforms Bayesian inference computations (very large summations or integrals) to an optimization problem.

- Inference by minimizing FE, also known as variational inference, is fully consistent with the "Method of Maximum Relative Entropy", which is by design the rational way to update beliefs from priors to posteriors when new information becomes available. Thus, FE mimimization is a "correct" inference procedure that generalizes Bayes rule.

- In general, global FE minimization is an unsolved problem and finding good local minima is at the heart of current Bayesian technology research. Three simplifying constraints on the posterior $q(z)$ in the FE functional are currently popular in practical settings:
    - mean-field assumptions
    - assuming a parametric from for $q$
    - EM algorithm

- These constraints often makes FE minimization implementable at the price of obtaining approximately Bayesian inference results.

- The back-ends of Probabilistic Programming languages often contain lots of methods for constrained FE minimization.

## OPTIONAL SLIDES

### FE Minimization by the Expectation-Maximization (EM) Algorithm

- The EM algorithm is a special case of FE minimization that focusses on Maximum-Likelihood estimation for models with latent variables.
- Consider a model $$p(x,z,\theta)$$ with observations $x = \{x_n\}$, latent variables $z=\{z_n\}$ and parameters $\theta$.

- We can write the following FE functional for this model:
$$\begin{align*}
F[q] =  \sum_z \sum_\theta q(z,\theta) \log \frac{q(z,\theta)}{p(x,z,\theta)}
\end{align*}$$

- The EM algorithm makes the following simplifying assumptions:
  1. The prior for the parameters is uninformative (uniform). This implies that
  $$p(x,z,\theta) = p(x,z|\theta) p(\theta) \propto p(x,z|\theta)$$
  2. A factorization constraint $$q(z,\theta) = q(z) q(\theta)$$
  3. The posterior for the parameters is a delta function:
  $$q(\theta) = \delta(\theta - \hat{\theta})$$

- Basically, these three assumptions turn FE minimization into maximum likelihood estimation for the parameters $\theta$ and the FE simplifies to
$$\begin{align*}
F[q,\theta] =  \sum_z q(z) \log \frac{q(z)}{p(x,z|\theta)}
\end{align*}$$

- The EM algorithm minimizes this FE by iterating (iteration counter: $i$) over

\begin{equation*}
\boxed{
\begin{aligned}
\mathcal{L}^{(i)}(\theta) &= \sum_z \overbrace{p(z|x,\theta^{(i-1)})}^{q^{(i)}(z)}  \log p(x,z|\theta) \quad &&\text{the E-step} \\
\theta^{(i)} &= \arg\max_\theta \mathcal{L}^{(i)}(\theta) &&\text{the M-step}
\end{aligned}}
\end{equation*}

- These choices are optimal for the given FE functional. In order to see this, consider the two decompositions
$$\begin{align}
F[q,\theta] &= \underbrace{-\sum_z q(z) \log p(x,z|\theta)}_{\text{energy}} - \underbrace{\sum_z q(z) \log \frac{1}{q(z)}}_{\text{entropy}} \tag{EE}\\
  &= \underbrace{\sum_z q(z) \log \frac{q(z)}{p(z|x,\theta)}}_{\text{divergence}} - \underbrace{\log p(x|\theta)}_{\text{log-likelihood}}  \tag{DE}
\end{align}$$

- The DE decomposition shows that the FE is minimized for the choice $q(z) := p(z|x,\theta)$. Also, for this choice, the FE equals the (negative) log-evidence (, which is this case simplifies to the log-likelihood).

- The EE decomposition shows that the FE is minimized wrt $\theta$ by minimizing the energy term. The energy term is computed in the E-step and optimized in the M-step.
  - Note that in the EM literature, the energy term is often called the _expected complete-data log-likelihood_.)

- In order to execute the EM algorithm, it is assumed that we can analytically execute the E- and M-steps. For a large set of models (including models whose distributions belong to the exponential family of distributions), this is indeed the case and hence the large popularity of the EM algorithm.

- The EM algorihm imposes rather severe assumptions on the FE (basically approximating Bayesian inference by maximum likelihood estimation). Over the past few years, the rise of Probabilistic Programming languages has dramatically increased the range of models for which the parameters can by estimated autmatically by (approximate) Bayesian inference, so the popularity of EM is slowly waning. (More on this in the Probabilistic Programming lessons).

- Bishop (2006) works out EM for the GMM in section 9.2.

### Code Example: EM-algorithm for the GMM on the Old-Faithful data set

We'll perform clustering on the data set from the [illustrative example](#illustrative-example) by fitting a GMM consisting of two Gaussians using the EM algorithm.

```julia
using Pkg; Pkg.activate("probprog/workspace");Pkg.instantiate();
IJulia.clear_output();

using DataFrames, CSV, LinearAlgebra
include("scripts/gmm_plot.jl") # Holds plotting function
old_faithful = CSV.read("datasets/old_faithful.csv", DataFrame);

X =  Array(Matrix{Float64}(old_faithful)')
N = size(X, 2)

# Initialize the GMM. We assume 2 clusters.
clusters = [MvNormal([4.;60.], [.5 0;0 10^2]);
            MvNormal([2.;80.], [.5 0;0 10^2])];
π_hat = [0.5; 0.5]                    # Mixing weights
γ = fill!(Matrix{Float64}(undef,2,N), NaN)  # Responsibilities (row per cluster)

# Define functions for updating the parameters and responsibilities
function updateResponsibilities!(X, clusters, π_hat, γ)
    # Expectation step: update γ
    norm = [pdf(clusters[1], X) pdf(clusters[2], X)] * π_hat
    γ[1,:] = (π_hat[1] * pdf(clusters[1],X) ./ norm)'
    γ[2,:] = 1 .- γ[1,:]
end
function updateParameters!(X, clusters, π_hat, γ)
    # Maximization step: update π_hat and clusters using ML estimation
    m = sum(γ, dims=2)
    π_hat = m / N
    μ_hat = (X * γ') ./ m'
    for k=1:2
        Z = (X .- μ_hat[:,k])
        Σ_k = Symmetric(((Z .* (γ[k,:])') * Z') / m[k])
        clusters[k] = MvNormal(μ_hat[:,k], convert(Matrix, Σ_k))
    end
end

# Execute the algorithm: iteratively update parameters and responsibilities
subplot(2,3,1); plotGMM(X, clusters, γ); title("Initial situation")
updateResponsibilities!(X, clusters, π_hat, γ)
subplot(2,3,2); plotGMM(X, clusters, γ); title("After first E-step")
updateParameters!(X, clusters, π_hat, γ)
subplot(2,3,3); plotGMM(X, clusters, γ); title("After first M-step")
iter_counter = 1
for i=1:3
    for j=1:i+1
        updateResponsibilities!(X, clusters, π_hat, γ)
        updateParameters!(X, clusters, π_hat, γ)
        iter_counter += 1
    end
    subplot(2,3,3+i);
    plotGMM(X, clusters, γ);
    title("After $(iter_counter) iterations")
end
PyPlot.tight_layout()
```

Note that you can step through the interactive demo yourself by running [this script](https://github.com/bertdv/AIP-5SSB0/blob/master/lessons/notebooks/scripts/interactive_em_demo.jl) in julia. You can run a script in julia by
`julia> include("path/to/script-name.jl")`

### Message Passing for Free Energy Minimization

- The Sum-Product (SP) update rule implements perfect Bayesian inference.
- Sometimes, the SP update rule is not analytically solvable.
- Fortunately, for many well-known Bayesian approximation methods, a message passing update rule can be created, e.g. [Variational Message Passing](https://en.wikipedia.org/wiki/Variational_message_passing) (VMP) for variational inference.
- In general, all of these message passing algorithms can be interpreted as minimization of a constrained free energy (e.g., see [Zhang et al. (2017)](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Zhang-2017-Unifying-Message-Passing-Algorithms.pdf), and hence these message passing schemes comply with [Caticha's Method of Maximum Relative Entropy](https://arxiv.org/abs/1011.0723), which, as discussed in the [variational Bayes lesson](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/Latent-Variable-Models-and-VB.ipynb) is the proper way for updating beliefs.
- Different message passing updates rules can be combined to get a hybrid inference method in one model.

### The Local Free Energy in a Factor Graph

- Consider an edge $x_j$ in a Forney-style factor graph for a generative model $p(x) = p(x_1,x_2,\ldots,x_N)$.

- Assume that the graph structure (factorization) is specified by
$$
p(x) = \prod_{a=1}^M p_a(x_a)
$$
where $a$ is a set of indices.
- Also, we assume a mean-field approximation for the posterior:
$$
q(x) = \prod_{i=1}^N q_i(x_i)
$$
and consequently a corresponding free energy functional
$$\begin{align*}
F[q] &= \sum_x q(x) \log \frac{q(x)}{p(x)} \\
  &= \sum_i \sum_{x_i} \left(\prod_{i=1}^N q_i(x_i)\right) \log \frac{\prod_{i=1}^N q_i(x_i)}{\prod_{a=1}^M p_a(x_a)}
\end{align*}$$

- With these assumptions, it can be shown that the FE evaluates to (exercise)
$$
F[q] = \sum_{a=1}^M \underbrace{\sum_{x_a} \left( \prod_{j\in N(a)} q_j(x_j)\cdot \left(-\log p_a(x_a)\right) \right) }_{\text{node energy }U[p_a]} - \sum_{i=1}^N \underbrace{\sum_{x_i} q_i(x_i) \log \frac{1}{q_i(x_i)}}_{\text{edge entropy }H[q_i]}
$$

- In words, the FE decomposes into a sum of (expected) energies for the nodes minus the entropies on the edges.

### Variational Message Passing

- Let us now consider the local free energy that is associated with edge corresponding to $x_j$.

- Apparently (see previous slide), there are three contributions to the free energy for $x_j$:
    - one entropy term for the edge $x_j$
    - two energy terms: one for each node that attaches to $x_j$ (in the figure: nodes $p_a$ and $p_b$)

- The local free energy for $x_j$ can be written as (exercise)
  $$
  F[q_j] \propto \sum_{x_j} q(x_j) \log \frac{q_j(x_j)}{\nu_a(x_j)\cdot \nu_b(x_j)}
  $$
  where
  $$\begin{align*}
  \nu_a(x_j) &\propto \exp\left( \mathbb{E}_{q_{k}}\left[ \log p_a(x_a)\right]\right) \\
  \nu_b(x_j) &\propto \exp\left( \mathbb{E}_{q_{l}}\left[ \log p_b(x_b)\right]\right)
  \end{align*}$$
  and $\mathbb{E}_{q_{k}}\left[\cdot\right]$ is an expectation w.r.t. all $q(x_k)$ with $k \in N(a)\setminus {j}$.

- $\nu_a(x_j)$ and $\nu_b(x_j)$  can be locally computed in nodes $a$ and $b$ respectively and can be interpreted as colliding messages over edge $x_j$.

- Local free energy minimization is achieved by setting
  $$
  q_j(x_j) \propto \nu_a(x_j) \cdot \nu_b(x_j)
  $$

- Note that message $\nu_a(x_j)$ depends on posterior beliefs over incoming edges ($k$) for node $a$, and in turn, the message from node $a$ towards edge $x_k$ depends on the belief $q_j(x_j)$. I.o.w., direct mutual dependencies exist between posterior beliefs over edges that attach to the same node.

- These considerations lead to the [Variational Message Passing](https://en.wikipedia.org/wiki/Variational_message_passing) procedure, which is an iterative free energy minimization procedure that can be executed completely through locally computable messages.

- Procedure VMP, see [Dauwels (2007), section 3](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Dauwels-2007-on-variational-message-passing-on-factor-graphs)
  > 1. Initialize all messages $q$ and $ν$, e.g., $q(\cdot) \propto 1$ and $\nu(\cdot) \propto 1$.

  > 2. Select an edge $z_k$ in the factor graph of $f(z_1,\ldots,z_m)$.

  > 3. Compute the two messages $\overrightarrow{\nu}(z_k)$ and $\overleftarrow{\nu}(z_k)$ by applying the following generic rule:
  $$
  \overrightarrow{\nu}(y) \propto \exp\left( \mathbb{E}_{q}\left[ \log g(x_1,\dots,x_n,y)\right] \right)
  $$
  > 4. Compute the marginal $q(z_k)$
  $$
  q(z_k) \propto \overrightarrow{\nu}(z_k) \overleftarrow{\nu}(z_k)
  $$
  and send it to the two nodes connected to the edge $x_k$.

  > 5. Iterate 2–4 until convergence.

### The Bethe Free Energy and Belief Propagation

- We showed that, under mean field assumptions, the FE can be decomposed into a sum of local FE contributions for the nodes ($a$) and edges ($i$):
$$
F[q] = \sum_{a=1}^M \underbrace{\sum_{x_a} \left( \prod_{j\in N(a)} q_j(x_j)\cdot \left(-\log p_a(x_a)\right) \right) }_{\text{node energy }U[p_a]} - \sum_{i=1}^N \underbrace{\sum_{x_i} q_i(x_i) \log \frac{1}{q_i(x_i)}}_{\text{edge entropy }H[q_i]}
$$

- The mean field assumption is very strong and may lead to large inference costs ($\mathrm{KL}(q(x),p(x|\text{data}))$). A more relaxed assumption is to allow joint posterior beliefs over the variables that attach to a node. This idea is expressed by the Bethe Free Energy:
$$
F_B[q] = \sum_{a=1}^M \left( \sum_{x_a} q_a(x_a) \log \frac{q_a(x_a)}{p_a(x_a)} \right)  - \sum_{i=1}^N (d_i - 1) \sum_{x_i} q_i(x_i) \log {q_i(x_i)}
$$
where $q_a(x_a)$ is the posterior joint belief over the variables $x_a$ (i.e., the set of variables that attach to node $a$), $q_i(x_i)$ is the posterior marginal belief over the variable $x_i$ and $d_i$ is the number of factor nodes that link to edge $i$. Moreover, $q_a(x_a)$ and $q_i(x_i)$ are constrained to obey the following equalities:
$$
  \sum_{x_a \backslash x_i} q_a(x_a) = q_i(x_i), ~~~ \forall i, \forall a \\
  \sum_{x_i} q_i(x_i) = 1, ~~~ \forall i \\
  \sum_{x_a} q_a(x_a) = 1, ~~~ \forall a \\
$$

- We form the Lagrangian by augmenting the Bethe Free Energy functional with the constraints:
$$
L[q] = F_B[q] + \sum_i\sum_{a \in N(i)} \lambda_{ai}(x_i) \left(q_i(x_i) - \sum_{x_a\backslash x_i} q(x_a) \right) + \sum_{i} \gamma_i \left(  \sum_{x_i}q_i(x_i) - 1\right) + \sum_{a}\gamma_a \left(  \sum_{x_a}q_a(x_a) -1\right)
$$

- The stationary solutions for this Lagrangian are given by
$$
q_a(x_a) = f_a(x_a) \exp\left(\gamma_a -1 + \sum_{i \in N(a)} \lambda_{ai}(x_i)\right) \\
q_i(x_i) = \exp\left(1- \gamma_i + \sum_{a \in N(i)} \lambda_{ai}(x_i)\right) ^{\frac{1}{d_i - 1}}
$$
where $N(i)$ denotes the factor nodes that have $x_i$ in their arguments and $N(a)$ denotes the set of variables in the argument of $f_a$.

- Stationary solutions are functions of Lagrange multipliers. This means that Lagrange multipliers need to be determined. Lagrange multipliers can be determined by plugging the stationary solutions back into the constraint specification and solving for the multipliers which ensure that the constraint is satisfied. The first constraint we consider is normalization, which yields the following identification:
$$
\gamma_a = 1 - \log \Bigg(\sum_{x_a}f_a(x_a)\exp\left(\sum_{i \in N(a)}\lambda_{ai}(x_i)\right)\Bigg)\\
\gamma_i = 1 + (d_i-1) \log\Bigg(\sum_{x_i}\exp\left( \frac{1}{d_i-1}\sum_{a \in N(i)} \lambda_{ai}(x_i)\right)\Bigg).
$$

- The functional form of the Lagrange multipliers that corresponds to the normalization constraint enforces us to obtain the Lagrange multipliers that correspond to the marginalization constraint. To do so we solve for
$$ \sum_{x_a \backslash x_i} f_a(x_a) \exp\left(\sum_{i \in N(a)} \lambda_{ai}(x_i)\right) = \exp\left(\sum_{a \in N(i)} \lambda_{ai}(x_i)\right) ^{\frac{1}{d_i - 1}} \\ \exp\left(\lambda_{ai}(x_i)\right)\sum_{x_a \backslash x_i} f_a(x_a) \exp\Bigg(\sum_{\substack{{j \in N(a)} \\ j \neq i}}\lambda_{aj}(x_j)\Bigg) = \exp\left(\sum_{a \in N(i)} \lambda_{ai}(x_i)\right) ^{\frac{1}{d_i - 1}}\\ \exp\left(\lambda_{ai}(x_i) + \lambda_{ia}(x_i)\right) = \exp\left(\sum_{a \in N(i)} \lambda_{ai}(x_i)\right) ^{\frac{1}{d_i - 1}}\,, $$
where we defined an auxilary function
$$
\exp(\lambda_{ia}(x_i)) \triangleq \sum_{x_a \backslash x_i} f_a(x_a) \exp\Bigg(\sum_{\substack{{j \in N(a)} \\ j \neq i}}\lambda_{aj}(x_j)\Bigg) \,.
$$
This definition is valid since it can be inverted by the relation
$$
\lambda_{ia}(x_i) = \frac{2-d_i}{d_i - 1}\lambda_{ai}(x_i) + \frac{1}{d_i -1}\sum_{\substack{c \in N(i)\\c \neq a}}\lambda_{ci}(x_i)
$$

- In general it is not possible to solve for the Lagrange multipliers analytically and we resort to iteratively obtaining the solutions. This leads to the **Belief Propagation algorithm** where the exponentiated Lagrange multipliers (messages) are updated iteratively via
$$ \mu_{ia}^{(k+1)}(x_i) = \sum_{x_a \backslash x_i} f_a(x_a) \prod_{\substack{{j \in N(a)} \\ j \neq i}}\mu^{(k)}_{aj}(x_j) \\ \mu_{ai}^{(k)}(x_i) = \prod_{\substack{c \in N(i) \\ c \neq a}}\mu^{(k)}_{ic}(x_i)\,, $$
where $k$ denotes iteration number and the messages are defined as
$$
\mu_{ia}(x_i) \triangleq \exp(\lambda_{ia}(x_i))\\
\mu_{ai}(x_i) \triangleq \exp(\lambda_{ai}(x_i))\,.
$$

- For a more complete overview of message passing as Bethe Free Energy minimization, see Zhang (2017).

```julia
open("../../styles/aipstyle.html") do f
    display("text/html", read(f, String))
end
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>

```julia

```


# Spectrum

[Colour](http://en.wikipedia.org/wiki/Colour) is defined as the characteristic of visual perception that can be described by attributes of hue, brightness (or lightness) and colourfulness (or saturation or chroma).

When necessary, to avoid confusion between other meanings of the word, the term "perceived colour" may be used.

Perceived colour depends on the spectral distribution of the colour stimulus, on the size, shape, structure and surround of the stimulus area, on the state of adaptation of the observer's visual system, and on the observer's experience of the prevailing and similar situations of observation. [1]

[Light](http://en.wikipedia.org/wiki/Light) is the electromagnetic radiation that is considered from the point of view of its ability to excite the human visual system. [2]

The portion of the electromatic radiation frequencies perceived in the approximate wavelength range 360-780 nanometres (nm) is called the [visible spectrum](http://en.wikipedia.org/wiki/Visible_spectrum).

```python
%matplotlib inline
```

```python
import colour
from colour.plotting import *

colour.filter_warnings(True, False)

colour_plotting_defaults()

# Plotting the visible spectrum.
visible_spectrum_plot()
```

The spectrum is defined as the display or specification of the monochromatic components of the radiation considered. [3]

At the core of [Colour](https://github.com/colour-science/colour/) is the *colour.colorimetry* sub-package, it defines the objects needed for spectral related computations and many others:

```python
from pprint import pprint

import colour.colorimetry as colorimetry

pprint(colorimetry.__all__)
```

    ['SpectralShape',
     'SpectralPowerDistribution',
     'MultiSpectralPowerDistribution',
     'DEFAULT_SPECTRAL_SHAPE',
     'constant_spd',
     'zeros_spd',
     'ones_spd',
     'blackbody_spd',
     'blackbody_spectral_radiance',
     'planck_law',
     'LMS_ConeFundamentals',
     'RGB_ColourMatchingFunctions',
     'XYZ_ColourMatchingFunctions',
     'CMFS',
     'LMS_CMFS',
     'RGB_CMFS',
     'STANDARD_OBSERVERS_CMFS',
     'ILLUMINANTS',
     'D_ILLUMINANTS_S_SPDS',
     'HUNTERLAB_ILLUMINANTS',
     'ILLUMINANTS_RELATIVE_SPDS',
     'LIGHT_SOURCES',
     'LIGHT_SOURCES_RELATIVE_SPDS',
     'LEFS',
     'PHOTOPIC_LEFS',
     'SCOTOPIC_LEFS',
     'BANDPASS_CORRECTION_METHODS',
     'bandpass_correction',
     'bandpass_correction_Stearns1988',
     'D_illuminant_relative_spd',
     'CIE_standard_illuminant_A_function',
     'mesopic_luminous_efficiency_function',
     'mesopic_weighting_function',
     'LIGHTNESS_METHODS',
     'lightness',
     'lightness_Glasser1958',
     'lightness_Wyszecki1963',
     'lightness_CIE1976',
     'lightness_Fairchild2010',
     'lightness_Fairchild2011',
     'LUMINANCE_METHODS',
     'luminance',
     'luminance_Newhall1943',
     'luminance_ASTMD153508',
     'luminance_CIE1976',
     'luminance_Fairchild2010',
     'luminance_Fairchild2011',
     'dominant_wavelength',
     'complementary_wavelength',
     'excitation_purity',
     'colorimetric_purity',
     'luminous_flux',
     'luminous_efficiency',
     'luminous_efficacy',
     'RGB_10_degree_cmfs_to_LMS_10_degree_cmfs',
     'RGB_2_degree_cmfs_to_XYZ_2_degree_cmfs',
     'RGB_10_degree_cmfs_to_XYZ_10_degree_cmfs',
     'LMS_2_degree_cmfs_to_XYZ_2_degree_cmfs',
     'LMS_10_degree_cmfs_to_XYZ_10_degree_cmfs',
     'SPECTRAL_TO_XYZ_METHODS',
     'spectral_to_XYZ',
     'ASTME30815_PRACTISE_SHAPE',
     'lagrange_coefficients_ASTME202211',
     'tristimulus_weighting_factors_ASTME202211',
     'adjust_tristimulus_weighting_factors_ASTME30815',
     'spectral_to_XYZ_integration',
     'spectral_to_XYZ_tristimulus_weighting_factors_ASTME30815',
     'spectral_to_XYZ_ASTME30815',
     'wavelength_to_XYZ',
     'WHITENESS_METHODS',
     'whiteness',
     'whiteness_Berger1959',
     'whiteness_Taube1960',
     'whiteness_Stensby1968',
     'whiteness_ASTME313',
     'whiteness_Ganz1979',
     'whiteness_CIE2004',
     'YELLOWNESS_METHODS',
     'yellowness',
     'yellowness_ASTMD1925',
     'yellowness_ASTME313']

> Note: *colour.colorimetry* sub-package public API is directly available from *colour* namespace.

[Colour](https://github.com/colour-science/colour/) computations are based on a comprehensive dataset available in pretty much each sub-packages, for example *colour.colorimetry.dataset* defines the following data:

```python
import colour.colorimetry.dataset as dataset

pprint(dataset.__all__)
```

    ['CMFS',
     'LMS_CMFS',
     'RGB_CMFS',
     'STANDARD_OBSERVERS_CMFS',
     'ILLUMINANTS',
     'D_ILLUMINANTS_S_SPDS',
     'HUNTERLAB_ILLUMINANTS',
     'ILLUMINANTS_RELATIVE_SPDS',
     'LIGHT_SOURCES',
     'LIGHT_SOURCES_RELATIVE_SPDS',
     'LEFS',
     'PHOTOPIC_LEFS',
     'SCOTOPIC_LEFS']

> Note: *colour.colorimetry.dataset* sub-package public API is directly available from *colour* namespace.

## Spectral Power Distribution

Whether it be a sample spectral power distribution, colour matching functions or illuminants, spectral data is manipulated using an object built with the *colour.SpectralPowerDistribution* class or based on it:

```python
import colour

# Defining a sample spectral power distribution data.
sample_spd_data = {
    380: 0.048,
    385: 0.051,
    390: 0.055,
    395: 0.06,
    400: 0.065,
    405: 0.068,
    410: 0.068,
    415: 0.067,
    420: 0.064,
    425: 0.062,
    430: 0.059,
    435: 0.057,
    440: 0.055,
    445: 0.054,
    450: 0.053,
    455: 0.053,
    460: 0.052,
    465: 0.052,
    470: 0.052,
    475: 0.053,
    480: 0.054,
    485: 0.055,
    490: 0.057,
    495: 0.059,
    500: 0.061,
    505: 0.062,
    510: 0.065,
    515: 0.067,
    520: 0.070,
    525: 0.072,
    530: 0.074,
    535: 0.075,
    540: 0.076,
    545: 0.078,
    550: 0.079,
    555: 0.082,
    560: 0.087,
    565: 0.092,
    570: 0.100,
    575: 0.107,
    580: 0.115,
    585: 0.122,
    590: 0.129,
    595: 0.134,
    600: 0.138,
    605: 0.142,
    610: 0.146,
    615: 0.150,
    620: 0.154,
    625: 0.158,
    630: 0.163,
    635: 0.167,
    640: 0.173,
    645: 0.180,
    650: 0.188,
    655: 0.196,
    660: 0.204,
    665: 0.213,
    670: 0.222,
    675: 0.231,
    680: 0.242,
    685: 0.251,
    690: 0.261,
    695: 0.271,
    700: 0.282,
    705: 0.294,
    710: 0.305,
    715: 0.318,
    720: 0.334,
    725: 0.354,
    730: 0.372,
    735: 0.392,
    740: 0.409,
    745: 0.420,
    750: 0.436,
    755: 0.450,
    760: 0.462,
    765: 0.465,
    770: 0.448,
    775: 0.432,
    780: 0.421}

spd = colour.SpectralPowerDistribution(sample_spd_data, name='Sample')
print(spd)
```

    [[  3.80000000e+02   4.80000000e-02]
     [  3.85000000e+02   5.10000000e-02]
     [  3.90000000e+02   5.50000000e-02]
     [  3.95000000e+02   6.00000000e-02]
     [  4.00000000e+02   6.50000000e-02]
     [  4.05000000e+02   6.80000000e-02]
     [  4.10000000e+02   6.80000000e-02]
     [  4.15000000e+02   6.70000000e-02]
     [  4.20000000e+02   6.40000000e-02]
     [  4.25000000e+02   6.20000000e-02]
     [  4.30000000e+02   5.90000000e-02]
     [  4.35000000e+02   5.70000000e-02]
     [  4.40000000e+02   5.50000000e-02]
     [  4.45000000e+02   5.40000000e-02]
     [  4.50000000e+02   5.30000000e-02]
     [  4.55000000e+02   5.30000000e-02]
     [  4.60000000e+02   5.20000000e-02]
     [  4.65000000e+02   5.20000000e-02]
     [  4.70000000e+02   5.20000000e-02]
     [  4.75000000e+02   5.30000000e-02]
     [  4.80000000e+02   5.40000000e-02]
     [  4.85000000e+02   5.50000000e-02]
     [  4.90000000e+02   5.70000000e-02]
     [  4.95000000e+02   5.90000000e-02]
     [  5.00000000e+02   6.10000000e-02]
     [  5.05000000e+02   6.20000000e-02]
     [  5.10000000e+02   6.50000000e-02]
     [  5.15000000e+02   6.70000000e-02]
     [  5.20000000e+02   7.00000000e-02]
     [  5.25000000e+02   7.20000000e-02]
     [  5.30000000e+02   7.40000000e-02]
     [  5.35000000e+02   7.50000000e-02]
     [  5.40000000e+02   7.60000000e-02]
     [  5.45000000e+02   7.80000000e-02]
     [  5.50000000e+02   7.90000000e-02]
     [  5.55000000e+02   8.20000000e-02]
     [  5.60000000e+02   8.70000000e-02]
     [  5.65000000e+02   9.20000000e-02]
     [  5.70000000e+02   1.00000000e-01]
     [  5.75000000e+02   1.07000000e-01]
     [  5.80000000e+02   1.15000000e-01]
     [  5.85000000e+02   1.22000000e-01]
     [  5.90000000e+02   1.29000000e-01]
     [  5.95000000e+02   1.34000000e-01]
     [  6.00000000e+02   1.38000000e-01]
     [  6.05000000e+02   1.42000000e-01]
     [  6.10000000e+02   1.46000000e-01]
     [  6.15000000e+02   1.50000000e-01]
     [  6.20000000e+02   1.54000000e-01]
     [  6.25000000e+02   1.58000000e-01]
     [  6.30000000e+02   1.63000000e-01]
     [  6.35000000e+02   1.67000000e-01]
     [  6.40000000e+02   1.73000000e-01]
     [  6.45000000e+02   1.80000000e-01]
     [  6.50000000e+02   1.88000000e-01]
     [  6.55000000e+02   1.96000000e-01]
     [  6.60000000e+02   2.04000000e-01]
     [  6.65000000e+02   2.13000000e-01]
     [  6.70000000e+02   2.22000000e-01]
     [  6.75000000e+02   2.31000000e-01]
     [  6.80000000e+02   2.42000000e-01]
     [  6.85000000e+02   2.51000000e-01]
     [  6.90000000e+02   2.61000000e-01]
     [  6.95000000e+02   2.71000000e-01]
     [  7.00000000e+02   2.82000000e-01]
     [  7.05000000e+02   2.94000000e-01]
     [  7.10000000e+02   3.05000000e-01]
     [  7.15000000e+02   3.18000000e-01]
     [  7.20000000e+02   3.34000000e-01]
     [  7.25000000e+02   3.54000000e-01]
     [  7.30000000e+02   3.72000000e-01]
     [  7.35000000e+02   3.92000000e-01]
     [  7.40000000e+02   4.09000000e-01]
     [  7.45000000e+02   4.20000000e-01]
     [  7.50000000e+02   4.36000000e-01]
     [  7.55000000e+02   4.50000000e-01]
     [  7.60000000e+02   4.62000000e-01]
     [  7.65000000e+02   4.65000000e-01]
     [  7.70000000e+02   4.48000000e-01]
     [  7.75000000e+02   4.32000000e-01]
     [  7.80000000e+02   4.21000000e-01]]

The sample spectral power distribution can be easily plotted against the visible spectrum:

```python
# Plotting the sample spectral power distribution.
single_spd_plot(spd)
```

With the sample spectral power distribution defined, we can retrieve its shape:

```python
# Displaying the sample spectral power distribution shape.
print(spd.shape)
```

    (380.0, 780.0, 5.0)

The shape returned is an instance of *colour.SpectralShape* class:

```python
repr(spd.shape)
```

    'SpectralShape(380.0, 780.0, 5.0)'

*colour.SpectralShape* is used throughout [Colour](https://github.com/colour-science/colour/) to define spectral dimensions and is instantiated as follows:

```python
# Using *colour.SpectralShape* with iteration.
shape = colour.SpectralShape(start=0, end=10, interval=1)
for wavelength in shape:
    print(wavelength)

# *colour.SpectralShape.range* method is providing the complete range of values.
shape = colour.SpectralShape(0, 10, 0.5)
shape.range()
```

    0.0
    1.0
    2.0
    3.0
    4.0
    5.0
    6.0
    7.0
    8.0
    9.0
    10.0

    array([  0. ,   0.5,   1. ,   1.5,   2. ,   2.5,   3. ,   3.5,   4. ,
             4.5,   5. ,   5.5,   6. ,   6.5,   7. ,   7.5,   8. ,   8.5,
             9. ,   9.5,  10. ])

[Colour](https://github.com/colour-science/colour/) defines three convenient objects to create constant spectral power distributions:

*  *colour.constant_spd*
*  *colour.zeros_spd*
*  *colour.ones_spd*

```python
# Defining a constant spectral power distribution.
constant_spd = colour.constant_spd(100)
print('"Constant Spectral Power Distribution"')
print(constant_spd.shape)
print(constant_spd[400])

# Defining a zeros filled spectral power distribution.
print('\n"Zeros Filled Spectral Power Distribution"')
zeros_spd = colour.zeros_spd()
print(zeros_spd.shape)
print(zeros_spd[400])

# Defining a ones filled spectral power distribution.
print('\n"Ones Filled Spectral Power Distribution"')
ones_spd = colour.ones_spd()
print(ones_spd.shape)
print(ones_spd[400])
```

    "Constant Spectral Power Distribution"
    (360.0, 780.0, 1.0)
    100.0

    "Zeros Filled Spectral Power Distribution"
    (360.0, 780.0, 1.0)
    0.0

    "Ones Filled Spectral Power Distribution"
    (360.0, 780.0, 1.0)
    1.0

By default the shape used by *colour.constant_spd*, *colour.zeros_spd* and *colour.ones_spd* is the one defined by *colour.DEFAULT_SPECTRAL_SHAPE* attribute using the *CIE 1931 2° Standard Observer* shape.

```python
print(repr(colour.DEFAULT_SPECTRAL_SHAPE))
```

    SpectralShape(360, 780, 1)

A custom shape can be passed to construct a constant spectral power distribution with tailored dimensions:

```python
colour.ones_spd(colour.SpectralShape(400, 700, 5))[450]
```

    1.0

Often interpolation of the spectral power distribution is needed, this is achieved with the *colour.SpectralPowerDistribution.interpolate* method. Depending on the wavelengths uniformity, the default interpolation method will differ. Following *CIE 167:2005* recommendation: The method developed by Sprague (1880) should be used for interpolating functions having a uniformly spaced independent variable and a *Cubic Spline* method for non-uniformly spaced independent variable. [4]

We can check the uniformity of the sample spectral power distribution:

```python
# Checking the sample spectral power distribution uniformity.
print(spd.is_uniform())
```

    True

Since the sample spectral power distribution is uniform the interpolation will be using the *colour.SpragueInterpolator* interpolator.

> Note: Interpolation happens in place and may alter your original data, use the *colour.SpectralPowerDistribution.clone* method to produce a copy of your spectral power distribution before interpolation.

```python
# Copying the sample spectral power distribution.
spd_copy = spd.copy()

# Interpolating the copied sample spectral power distribution.
spd_copy.interpolate(colour.SpectralShape(400, 770, 1))
spd_copy[401]
```

    0.065809599999999996

```python
# Comparing the interpolated spectral power distribution with the original one.
multi_spd_plot([spd, spd_copy], bounding_box=[730,780, 0.1, 0.5])
```

Extrapolation although dangerous can be used to help aligning two spectral power distributions together. *CIE 015:2004 Colorimetry, 3rd Edition* recommends that unmeasured values may be set equal to the nearest measured value of the appropriate quantity in truncation: [5]

```python
# Extrapolating the copied sample spectral power distribution.
spd_copy.extrapolate(colour.SpectralShape(340, 830))
spd_copy[340], spd_copy[830]
```

    (0.065000000000000002, 0.44800000000000018)

The underlying interpolator can be swapped for any of the *Colour* interpolators.

```python
pprint([
    export for export in colour.algebra.interpolation.__all__
    if 'Interpolator' in export
])
```

    [u'KernelInterpolator',
     u'LinearInterpolator',
     u'SpragueInterpolator',
     u'CubicSplineInterpolator',
     u'PchipInterpolator',
     u'NullInterpolator']

```python
# Changing interpolator while trimming the copied spectral power distribution.
spd_copy.interpolate(
    colour.SpectralShape(400, 700, 10), interpolator=colour.LinearInterpolator)
```

    SpectralPowerDistribution([[  4.00000000e+02,   6.50000000e-02],
                               [  4.10000000e+02,   6.80000000e-02],
                               [  4.20000000e+02,   6.40000000e-02],
                               [  4.30000000e+02,   5.90000000e-02],
                               [  4.40000000e+02,   5.50000000e-02],
                               [  4.50000000e+02,   5.30000000e-02],
                               [  4.60000000e+02,   5.20000000e-02],
                               [  4.70000000e+02,   5.20000000e-02],
                               [  4.80000000e+02,   5.40000000e-02],
                               [  4.90000000e+02,   5.70000000e-02],
                               [  5.00000000e+02,   6.10000000e-02],
                               [  5.10000000e+02,   6.50000000e-02],
                               [  5.20000000e+02,   7.00000000e-02],
                               [  5.30000000e+02,   7.40000000e-02],
                               [  5.40000000e+02,   7.60000000e-02],
                               [  5.50000000e+02,   7.90000000e-02],
                               [  5.60000000e+02,   8.70000000e-02],
                               [  5.70000000e+02,   1.00000000e-01],
                               [  5.80000000e+02,   1.15000000e-01],
                               [  5.90000000e+02,   1.29000000e-01],
                               [  6.00000000e+02,   1.38000000e-01],
                               [  6.10000000e+02,   1.46000000e-01],
                               [  6.20000000e+02,   1.54000000e-01],
                               [  6.30000000e+02,   1.63000000e-01],
                               [  6.40000000e+02,   1.73000000e-01],
                               [  6.50000000e+02,   1.88000000e-01],
                               [  6.60000000e+02,   2.04000000e-01],
                               [  6.70000000e+02,   2.22000000e-01],
                               [  6.80000000e+02,   2.42000000e-01],
                               [  6.90000000e+02,   2.61000000e-01],
                               [  7.00000000e+02,   2.82000000e-01]],
                              interpolator=SpragueInterpolator,
                              interpolator_args={},
                              extrapolator=Extrapolator,
                              extrapolator_args={u'right': None, u'method': u'Constant', u'left': None})

The extrapolation behaviour can be changed for *Linear* method instead of the *Constant* default method or even use arbitrary constant *left* and *right* values:

```python
# Extrapolating the copied sample spectral power distribution with *Linear* method.
spd_copy.extrapolate(
    colour.SpectralShape(340, 830),
    extrapolator_args={'method': 'Linear',
                       'right': 0})
spd_copy[340], spd_copy[830]
```

    (0.046999999999999348, 0.0)

Aligning a spectral power distribution is a convenient way to first interpolate the current data within its original bounds then if needed extrapolates any missing values to match the requested shape:

```python
# Aligning the cloned sample spectral power distribution.
# We first trim the spectral power distribution as above.
spd_copy.interpolate(colour.SpectralShape(400, 700))
spd_copy.align(colour.SpectralShape(340, 830, 5))
spd_copy[340], spd_copy[830]
```

    (0.065000000000000002, 0.28199999999999975)

The *colour.SpectralPowerDistribution* class also supports various arithmetic operations like *addition*, *subtraction*, *multiplication*, *division* or *exponentiation* with *numeric* and *array_like* variables or other *colour.SpectralPowerDistribution* class instances:

```python
spd = colour.SpectralPowerDistribution({
    410: 0.25,
    420: 0.50,
    430: 0.75,
    440: 1.0,
    450: 0.75,
    460: 0.50,
    480: 0.25
})

print((spd.copy() + 1).values)
print((spd.copy() * 2).values)
print((spd * [0.35, 1.55, 0.75, 2.55, 0.95, 0.65, 0.15]).values)
print((spd * colour.constant_spd(2, spd.shape) * colour.constant_spd(3, spd.shape)).values)
```

    [ 1.25  1.5   1.75  2.    1.75  1.5   1.25]
    [ 0.5  1.   1.5  2.   1.5  1.   0.5]
    [ 0.0875  0.775   0.5625  2.55    0.7125  0.325   0.0375]
    [ 1.5  3.   4.5  6.   4.5  3.   nan  1.5]

The spectral power distribution can be normalised with an arbitrary factor:

```python
print(spd.normalise().values)
print(spd.normalise(100).values)
```

    [ 0.25  0.5   0.75  1.    0.75  0.5   0.25]
    [  25.   50.   75.  100.   75.   50.   25.]

## Colour Matching Functions

In the late 1920's, Wright (1928) and Guild (1931) independently conducted a series of colour matching experiments to quantify the colour ability of an average human observer which laid the foundation for the specification of the [CIE XYZ colourspace](http://en.wikipedia.org/wiki/CIE_color_space#Definition_of_the_CIE_XYZ_color_space). The results obtained were summarized by the *Wright & Guild 1931 2° RGB CMFs* $\bar{r}(\lambda)$,$\bar{g}(\lambda)$,$\bar{b}(\lambda)$ colour matching functions: they represent the amounts of three monochromatic primary colours $\textbf{R}$,$\textbf{G}$,$\textbf{B}$ needed to match the test colour at a single wavelength of light.

> See Also: The [Colour Matching Functions](cmfs.ipynb) notebook for in-depth informations about the colour matching functions.

```python
# Plotting *Wright & Guild 1931 2 Degree RGB CMFs* colour matching functions.
single_cmfs_plot('Wright & Guild 1931 2 Degree RGB CMFs')
```

With an RGB model of human vision based on *Wright & Guild 1931 2° RGB CMFs* $\bar{r}(\lambda)$,$\bar{g}(\lambda)$,$\bar{b}(\lambda)$ colour matching functions and for pragmatic reasons the *CIE* members developed a new colour space that would relate to the *CIE RGB* colourspace but for which all tristimulus values would be positive for real colours: *CIE XYZ* described with $\bar{x}(\lambda)$,$\bar{y}(\lambda)$,$\bar{z}(\lambda)$ colour matching functions.

```python
# Plotting *CIE XYZ 1931 2 Degree Standard Observer* colour matching functions.
single_cmfs_plot('CIE 1931 2 Degree Standard Observer')
```

In the 1960's it appeared that cones were present in a larger region of eye than the one initially covered by the experiments that lead to the *CIE 1931 2° Standard Observer* specification.

As a result, colour computations done with the *CIE 1931 2° Standard Observer* do not always correlate to the visual observation.

In 1964, the *CIE* defined an additional standard observer: the *CIE 1964 10° Standard Observer* derived from the work of Stiles and Burch (1959), and Speranskaya (1959). The *CIE 1964 10° Standard Observer* is believed to be a better representation of the human vision spectral response and recommended when dealing with a field of view of more than 4°.

For example and as per *CIE* recommendation, the *CIE 1964 10° Standard Observer* is commonly used with spectrophotometers for colour measurements whereas colorimeters generally use the *CIE 1931 2° Standard Observer* for quality control and other colour evaluation applications.

## CIE XYZ Tristimulus Values

The *CIE XYZ* tristimulus values specify a colour stimulus in terms of the visual system. Their values for colour of a surface with spectral reflectance $\beta(\lambda)$ under an illuminant of relative spectral power $S(\lambda)$ are calculated using the following equations: [6]

$$
\begin{equation}
X=k\int_{\lambda}\beta(\lambda)S(\lambda)\bar{x}(\lambda)d\lambda\\
Y=k\int_{\lambda}\beta(\lambda)S(\lambda)\bar{y}(\lambda)d\lambda\\
Z=k\int_{\lambda}\beta(\lambda)S(\lambda)\bar{z}(\lambda)d\lambda
\end{equation}
$$
where
$$
\begin{equation}
k=\cfrac{100}{\int_{\lambda}S(\lambda)\bar{y}(\lambda)d\lambda}
\end{equation}
$$

However in virtually all practical computations of *CIE XYZ* tristimulus values, the integrals are replaced by summations:

$$
\begin{equation}
X=k\sum\limits_{\lambda=\lambda_a}^{\lambda_b}\beta(\lambda)S(\lambda)\bar{x}(\lambda)\Delta\lambda\\
Y=k\sum\limits_{\lambda=\lambda_a}^{\lambda_b}\beta(\lambda)S(\lambda)\bar{y}(\lambda)\Delta\lambda\\
Z=k\sum\limits_{\lambda=\lambda_a}^{\lambda_b}\beta(\lambda)S(\lambda)\bar{z}(\lambda)\Delta\lambda\\
\end{equation}
$$
where
$$
\begin{equation}
k=\cfrac{100}{\sum\limits_{\lambda=\lambda_a}^{\lambda_b}S(\lambda)\bar{y}(\lambda)\Delta\lambda}
\end{equation}
$$

Calculating the *CIE XYZ* tristimulus values of a colour stimulus is done using the *colour.spectral_to_XYZ* definition which follows *ASTM E2022–11* and *ASTM E308–15* practises computation method:

```python
spd = colour.SpectralPowerDistribution(sample_spd_data, name='Sample')
cmfs = colour.STANDARD_OBSERVERS_CMFS['CIE 1931 2 Degree Standard Observer']
illuminant = colour.ILLUMINANTS_RELATIVE_SPDS['A']

# Calculating the sample spectral power distribution *CIE XYZ* tristimulus values.
colour.spectral_to_XYZ(spd, cmfs, illuminant)
```

    array([ 14.78676625,  10.97815867,   1.99023459])

> Note: Output *CIE XYZ* colourspace matrix is in domain [0, 100].

*CIE XYZ* tristimulus values can be plotted into the *CIE 1931 Chromaticity Diagram*:

```python
import pylab

# Plotting the *CIE 1931 Chromaticity Diagram*.
# The argument *standalone=False* is passed so that the plot doesn't get displayed
# and can be used as a basis for other plots.
chromaticity_diagram_plot_CIE1931(standalone=False)

# Calculating the *xy* chromaticity coordinates.
# The output domain of *colour.spectral_to_XYZ* is [0, 100] and
# the input domain of *colour.XYZ_to_sRGB* is [0, 1].
# We need to take it in account and rescale the input *CIE XYZ* colourspace matrix.
x, y = colour.XYZ_to_xy(colour.spectral_to_XYZ(spd, cmfs, illuminant) / 100)

# Plotting the *xy* chromaticity coordinates.
pylab.plot(x, y, 'o-', color='white')

# Annotating the plot.
pylab.annotate(spd.name,
               xy=(x, y),
               xytext=(-50, 30),
               textcoords='offset points',
               arrowprops=dict(arrowstyle='->', connectionstyle='arc3, rad=-0.2'))

# Displaying the plot.
render(standalone=True)
```

Retrieving the *CIE XYZ* tristimulus values of any wavelength from colour matching functions is done using the *colour.wavelength_to_XYZ* definition, if the value requested is not available, the colour matching functions will be interpolated following *CIE 167:2005* recommendation:

```python
colour.wavelength_to_XYZ(546.1, colour.STANDARD_OBSERVERS_CMFS['CIE 1931 2 Degree Standard Observer'])
```

    array([ 0.3755316 ,  0.98444552,  0.01220285])

## Bibliography

1. ^ CIE. (n.d.). 17-198 colour (perceived). Retrieved June 26, 2014, from http://eilv.cie.co.at/term/198
2. ^ CIE. (n.d.). 17-659 light. Retrieved June 26, 2014, from http://eilv.cie.co.at/term/659
3. ^ CIE. (n.d.). 17-1238 spectrum. Retrieved June 27, 2014, from http://eilv.cie.co.at/term/1238
4. ^ CIE TC 1-38. (2005). 9. INTERPOLATION. In CIE 167:2005 Recommended Practice for Tabulating Spectral Data for Use in Colour Computations (pp. 14–19). ISBN:978-3-901-90641-1
5. ^ CIE TC 1-48. (2004). CIE 015:2004 Colorimetry, 3rd Edition. CIE 015:2004 Colorimetry, 3rd Edition (pp. 1–82). ISBN:978-3-901-90633-6
6. ^ Wyszecki, G., & Stiles, W. S. (2000). Integration Replace by Summation. In *Color Science: Concepts and Methods, Quantitative Data and Formulae* (pp. 158–163). Wiley. ISBN:978-0471399186


## First Assignment

#### 1) Apply the appropriate string methods to the **x** variable (as '.upper') to change it exactly to: "$Dichlorodiphenyltrichloroethane$".

```python
x = "DiClOrod  IFeNi  lTRicLOr oETaNo  DiChlorod iPHeny lTrichL oroEThaNe"

```

```python
print(x)

```

    DiClOrod  IFeNi  lTRicLOr oETaNo  DiChlorod iPHeny lTrichL oroEThaNe

```python
x = x.replace(' ', '').title()

```

```python
x = x[x.find('Dichlorodipheny') :]
```

```python
print(x)
```

    e

```python
x = x.split(',',31)
```

```python
print(x)
```

    e

```python
uppercase_string=x.upper()
```

```python
print(uppercase_string)
```

    DICLOROD  IFENI  LTRICLOR OETANO  DICHLOROD IPHENY LTRICHL OROETHANE

#### 2) Assign respectively the values: 'word', 15, 3.14 and 'list' to variables A, B, C and D in a single line of code. Then, print them in that same order on a single line separated by a space, using only one print statement.

```python
# 1
A,B,C,D = ['word', 15, 3.14, 'list']
```

```python
D
```

    'list'

#### 3) Use the **input()** function to receive an input in the form **'68.4 1.71'**, that is, two floating point numbers in a line separated by space. Then, assign these numbers to the variables **w** and **h** respectively, which represent an individual's weight and height (hint: take a look at the '.split()' method). With this data, calculate the individual's Body Mass Index (BMI) from the following relationship:

\begin{equation}
BMI = \dfrac{weight}{height^2}
\end{equation}

```python
#1
def bmi (weight = 0, height = 0):
    if bmi== 0:
        bmi = input('68.4' '1.71')
        bmi.split('','')
        bmi = float(weight)/((float(height)/100)**2)
        print(bmi)

```

```python
print(bmi)
```

    <function bmi at 0x7fc9360d1b90>

```python
w, h = 68.4, 1.71
print(w)
print(h)
```

    68.4
    1.71

```python
#2
```

```python
x = input('enter height space weight : ')
```

    enter height and weight : 168 68

```python
x
```

    '168 68'

```python
y = x.split(' ')
```

```python
y
```

    ['168', '68']

```python
height = float(y[0])
```

```python
type(height)
```

    float

#### This value can also be classified according to ranges of values, following to the table below. Use conditional structures to classify and print the classification assigned to the individual.

<\center>

(source: https://healthtravelguide.com/bmi-calculator/)

```python
#1
bmi = int(input())
def bmi(weight= 0, height= 0):
    if weight == 0:
        weight = input(w)
    if height == 0:
        height = input(h)
    if bmi  < 18.5 and bmi == 18.5:
       print('Underweight')
    if bmi  > 18.5 and bmi < 24.9 :
       print('Normal weight')
    if bmi > 25.0 and bmi ==25.0:
       print('Pre-obesity or obesity')
```

    2

```python
#2
bmi = int(input('BMI: '))
    if bmi < 18.5:
        print('Underweight')
    elif bmi > 18.5 and bmi < 24.9:
        print('Normal weight')
    elif bmi > 25.0 and bmi ==25.0:
        print('Pre-obesity or obesity')
```

```python
print(bmi)
```

#### 4) Receive an integer as an input and, using a loop, calculate the factorial of this number, that is, the product of all the integers from one to the number provided.

```python
for integer in range (1,10):
    print('   ')
    print(f'{integer} ')
    print()

```

    1

    2

    3

    4

    5

    6

    7

    8

    9

#### 5) Using a while loop and the input function, read an indefinite number of integers until the number read is -1. Present the sum of all these numbers in the form of a print, excluding the -1 read at the end.

```python
x = 1
while x < 1:
    print(x)
    x += -1
    print(do_sum(x)).pop
```

```python
print(x)
```

    1

#### 6) Read the **first name** of an employee, his **amount of hours worked** and the **salary per hour** in a single line separated by commas. Next, calculate the **total salary** for this employee and show it to two decimal places.

```python
# which data, should I invent name and salary?
record = {'first name': 'Susan', 'amount hours worked':150, 'Salary per hour':'12'}
print(record)
def total_salary(hour_worked, wage):
    hours_worked = 150
    wage = 12.00
    total_salary = [hours_worked]*[wage]
print(total_salary)
```

    {'first name': 'Susan', 'amount hours worked': 150, 'Salary per hour': '12'}
    <function total_salary at 0x7f6988501050>

#### 7) Read three floating point values **A**, **B** and **C** respectively. Then calculate itens a, b, c, d and e:

```python
import math
```

    a) the area of the triangle rectangle with A as the base and C as the height.

```python
A = 2
C = 1.5
from math import area
area = float(input('area of trinagle'('1/2' *A * C)
print(('area of trinagle ') + str(A) + 'is: ' str(1/2 *A*C))
```

    b) the area of the circle of radius C. (pi = 3.14159)

```python
from math import pi
r = float(input('area of the circle radius C : '))
print ('area of the circle radius C ' + str(r) + " is: " + str(pi * r**2))
```

    area of the circle radius C : 1
    area of the circle radius C 1.0 is: 3.141592653589793

    c) the area of the trapezoid that has A and B for bases and C for height.

```python
A = b1
B = b2
C = h
def area(b1, b2, h):
    return ((b1 + b2) / 2) * h
area = area(b1, b2, h)
print('area is:', area)

```

    area is: 4.275

    d) the area of the square that has side B.

```python
B = side
def areaSquare( side ):
    area = side * side
    return area

```

    e) the area of the rectangle that has sides A and B.

```python
A = l
b = b
l = float(input('Enter the length of a Rectangle: '))
b = float(input('Enter the breadth of a Rectangle: '))
Area = l * b
print("Area of a Rectangle is: %.2f" %Area)
l = A
b = B

```

    Enter the length of a Rectangle: 2
    Enter the breadth of a Rectangle: 5
    Area of a Rectangle is: 10.00

#### 8) Read **the values a, b and c** and calculate the **roots of the second degree equation** $ax^{2}+bx+c=0$ using [this formula](https://en.wikipedia.org/wiki/Quadratic_equation). If it is not possible to calculate the roots, display the message **“There are no real roots”**.

```python
import math
def equationroots (a, b, c):
    dis = (b * b - 4 * a * c)
    sqrt_val = math.sqrt(abs(dis))
    if dis > 0:
        print(' There are no real roots ')
        print((-b + sqrt_val)/(2 * a))
        print((-b - sqrt_val)/(2 * a))

```

#### 9) Read four floating point numerical values corresponding to the coordinates of two geographical coordinates in the cartesian plane. Each point will come in a line with its coordinates separated by space. Then calculate and show the distance between these two points.

(obs: $d=\sqrt{(x_1-x_2)^2 + (y_1-y_2)^2}$)

```python
point1 = [30.0443879, 31.2357257]
point2=  [41.8933203, 12.4829321]
point1 = (x1, x2)
point2 = (y1, y2)
float(input('Point 1: ', (point1))
float(input('Point 2: ', (point2))
distance = equationroots(x,y)
print(("Distance between two points is: ") %.2 f %2)
```

#### 10) Read **two floating point numbers** on a line that represent **coordinates of a cartesian point**. With this, use **conditional structures** to determine if you are at the origin, printing the message **'origin'**; in one of the axes, printing **'x axis'** or **'y axis'**; or in one of the four quadrants, printing **'q1'**, **'q2**', **'q3'** or **'q4'**.

```python
my_origin = [30.0443879, 31.2357257]
    if i == my_origin:
        print('origin')

```

#### 11) Read an integer that represents a phone code for international dialing.
#### Then, inform to which country the code belongs to, considering the generated table below:
(You just need to consider the first 10 entries)

```python
import pandas as pd
df = pd.read_html('https://en.wikipedia.org/wiki/Telephone_numbers_in_Europe')[1]
df = df.iloc[:,:2]
df.head(20)

```

      Country
      Country calling code

      0
      Austria
      43

      1
      Belgium
      32

      2
      Bulgaria
      359

      3
      Croatia
      385

      4
      Cyprus
      357

      5
      Czech Republic
      420

      6
      Denmark
      45

      7
      Estonia
      372

      8
      Finland
      358

      9
      France
      33

      10
      Germany
      49

      11
      Greece
      30

      12
      Hungary
      36

      13
      Iceland
      354

      14
      Ireland
      353

      15
      Italy
      39

      16
      Latvia
      371

      17
      Liechtenstein
      423

      18
      Lithuania
      370

      19
      Luxembourg
      352

```python
codes = {'43': 'Austria','32': 'Belgium','358': 'Finland','33': 'France','49': 'Germany','30': 'Greece', '36' : 'Hungary','354' : 'Iceland','353' : 'Ireland','39': 'Italy'}
```

```python
codes
```

    {'30': 'Greece',
     '32': 'Belgium',
     '33': 'France',
     '353': 'Ireland',
     '354': 'Iceland',
     '358': 'Finland',
     '36': 'Hungary',
     '39': 'Italy',
     '43': 'Austria',
     '49': 'Germany'}

```python
my_code = input('What is your country code? ')
```

    What is your country code? 33

```python
my_code
```

    '33'

```python
codes[my_code]
```

    'France'

#### 12) Write a piece of code that reads 6 numbers in a row. Next, show the number of positive values entered. On the next line, print the average of the values to one decimal place.

```python
my_new_list= ('5','7','15','1','12','2')
print(my_new_list)
```

    ('5', '7', '15', '1', '12', '2')

```python
my_new_list=int
sum(my_new_list)/6
```

#### 13) Read an integer **N**. Then print the **square of each of the even values**, from 1 to N, including N, if applicable, arranged one per line.

```python
N = i
int(i) for i in range(1,21)*2:
    print(i)

```

#### 14) Using **input()**, read an integer and print its classification as **'even / odd'** and **'positive / negative'** . The two classes for the number must be printed on the same line separated by a space. In the case of zero, print only **'null'**.

```python
my_new_integer = int(input())
if my_new_integer %2 and my_new_integer > 0:
    print('even positive')
elif my_new_integer == 0:
    print('null')
elif my_new_integer < 0 and my_new_integer %2 == 0:
    print('even negative')
elif my_new_integer > 0 and my_new_integer %2 != 0:
    print('odd positive')
else:
    print('odd negative')

```

    0
    null

## Challenge
#### 15) Ordering problems are recurrent in the history of programming. Over time, several algorithms have been developed to fulfill this function. The simplest of these algorithms is the [**Bubble Sort**](https://en.wikipedia.org/wiki/Bubble_sort), which is based on comparisons of elements two by two in a loop of passes through the elements. Your mission, if you decide to accept it, will be to input six whole numbers ramdonly ordered. Then implement the **Bubble Sort** principle to order these six numbers **using only loops and conditionals**.
#### At the end, print the six numbers in ascending order on a single line separated by spaces.

```python

```