# Lecture 25 - Variational Inference

## Objectives

+ to introduce variational inference as a class of techniques for approximate Bayesian Inference.
+ to use automatic differentiation variational inference (ADVI) for performing Bayesian inference, using `PyMC3`.

## References

+ Variational Inference: A Review for Statisticians (Blei et al, 2018) - https://arxiv.org/pdf/1601.00670.pdf.

+ Automatic Differentiation Variational Inference (Kucukelbir et al, 2016) - https://arxiv.org/pdf/1603.00788.pdf.

+ Autoencoding Variational Bayes (Kingma and Welling, 2014) - https://arxiv.org/pdf/1312.6114.pdf

+ Black Box Variational Inference (Ranganath et al, 2013) - https://arxiv.org/pdf/1401.0118.pdf.

## Additional Reading

+ Stein Variational Gradient Descent (Liu and Wang, 2016) (https://arxiv.org/pdf/1608.04471.pdf)

+ Variational Inference with Normalizing Flows (Rezende and Mohamed, 2016) - https://arxiv.org/pdf/1505.05770.pdf.

```python
import numpy as np
import theano as th
from theano import tensor as tt, shared
import pymc3 as pm

from scipy import stats as st
from scipy import optimize

import pandas as pd
import arviz as az
from graphviz import Digraph

import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
%matplotlib inline
```

## Bayesian Inference

### Quick Review

Once again, let's begin with a review of Bayesian inference.

Our goal is to derive a probability distribution over unknown quantities (or latent variables), conditional on any observed data (i.e. a posterior distribution).
Without loss of generality, we denote all unknown quantities in our model as $\theta$ and the observed data as $\mathcal{D}$.

We start with a description of our prior state of knowledge over $\theta$ - $p(\theta)$.

We then specify a conditional probabilistic model that links the observed data with the unknown quantities $p(\mathcal{D}|\theta)$ (the likelihood).
We want $p(\theta|\mathcal{D})$ which we know, from Bayes rule, to be:
$
p(\theta | \mathcal{D}) \propto p(\mathcal{D}, \theta).
$

The posterior distribution $p(\theta | \mathcal{D})$ captures our state of knowledge about $\theta$ conditional on all the information available to us $\mathcal{D}$.

In the Bayesian framework, predictions about unseen data (or test data), are posed as expectations over this posterior distribution.

### What is the problem?

Unfortunately, as you already know, the posterior distribution is more often than not unavailable in closed form.
This is due to the intractablity of the *evidence* (or *marginal likelihood*), i.e., the denominator in the Bayes' rule, $Z = \int p(\theta, \mathcal{D}) \mathrm{d}\theta$.
Infact, there are only a small class of prior-posterior models that admit closed form expressions for the posterior distributions (*conjugate models*).

### Approximating the posterior

There are several approaches to do this:

1. The posterior density $p(\theta | \mathcal{D})$ is approximated with a point mass density, i.e., $p(\theta | \mathcal{D})  = \delta_{\theta^*}(\theta)$, where, $\delta_{\theta^*}(\theta) = \begin{cases}  1, \text{ if } \theta = \theta^*, \\  0 \text{ otherwise.} \end{cases}$
This is the well-known *maximum a-posteriori* (MAP) estimation procedure. The parameter $\theta^*$ is obtained as the solution of the optimization problem, $\theta^* = \underset{\theta}{\mathrm{argmax}} p(\theta, \mathcal{D})$. The MAP approximation is often justified by the assumption that the true posterior distribution $p(\theta|\mathcal{D})$ has a single, sharply peaked mode. In practice this approach often provides reasonable predictive accuracy but is unable to capture any of the epistemic uncertainty induced by limited data.

2. The posterior distribution is approximated with a finite number of particles, i.e., $p(\theta | \mathcal{D}) = \sum_{i=1}^{N} w^{(i)} \delta (\theta - \theta^{(i)})$.  The most popular class of techniques that approximates the posterior distribution this way is Markov Chain Monte Carlo (MCMC). Recall that the general idea of MCMC is to construct a discrete-time, reversible and ergodic Markov Chain whose equilibrium distribution is the target posterior distribution. The goal is to simulate the Markov Chain long enough that it enters it's equilibrium phase (i.e. target posterior density). Once this is accomplished, sampling from the Markov Chain is the same as sampling from the target posterior density. Since MCMC samples (in theory) directly from the posterior, the weights of the approximation $, w^{(i)}$ are simply set to 1.
There are several other approaches to approximate probability densities with particle distributions such as Sequential Monte Carlo (SMC) (which developed primarily as tools for inferring latent variables in state-space models but can be used for general purpose inference) and Stein Variational Gradient Descent (SVGD).

3. Set up a parameterized family of densities over the latent variables - $q_{\phi}(\theta)$, and infer the parameters, $\phi$ by solving an optimization problem of the form:
$$
\phi^{*} = \underset{\phi}{\mathrm{argmin}} \ \mathrm{D}[ p(\theta| \mathcal{D}) , q_{\phi}(\theta)],
$$
where, $\mathrm{D}[\cdot, \cdot]$ is some measure of discrepancy between the approximate (or *variational*) posterior and the true posterior.
Needless to say, we want to set up this optimization problem such that we only need to know $p(\theta | \mathcal{D})$ upto a multiplicative constant.
*Variational Inference* (VI) is the name given to this general class of methods that seek to approximate the posterior this way.

### Variational Inference

Different VI procedures are obtained based on different choices of the approximating family $q_{\phi}(\cdot)$ and the functional $\mathrm{D}[\cdot, \cdot]$. The most standard choice for $\mathrm{D}$ is the *Kullback Leibler (KL) divergence*.
The KL divergence between two densities $q(\theta)$ and $p(\theta)$ is defined as follows:
$$
\mathrm{KL}[q(\theta)|| p(\theta)] = \int q(\theta) \log \left( \frac{q(\theta)}{p(\theta)} \right) \mathrm{d}\theta = \mathbb{E}_{q(\theta)} \left[ \log \left( \frac{q(\theta)}{p(\theta)} \right) \right].
$$
The KL divergence is always non-negative, i.e., $\mathrm{KL}[q(\theta)|| p(\theta)] \ge 0$, with $\mathrm{KL}[q(\theta)|| p(\theta)] = 0$ implying that $q(\theta) = p(\theta)$ *almost everywhere*.
Our inference goal can, therefore, be stated as follows - given a choice of a family of densities $q(\cdot)$, parameterized by $\phi$, what is the setting of $\phi$ that will return the closest match, i.e. minimum KL divergence, between the approximate posterior $q(\theta)$ and the true posterior $p(\theta|\mathcal{D})$?

This brings us to $q$ - the approximate posterior. Notice that we have made no assumptions on $q$ thus far. We can, ofcourse, pick any arbitrary distribution we want to approximate the posterior. However, in practice, we pick $q$ such that it satisfies some desirable properties:
1. If we know that a latent variable has finite support (positive reals for instance), we pick $q$ such $q$ itself has support on the same interval only.
2. We would also like $q$ to be easy to sample from and easy to evaluate it's log probability since the variational objective requires computing an expectation over log probability ratios. A common simplfying assumption that enables easier sampling and log probability computation is the *mean-field* assumption - i.e., setting up approximation such that the individual latent variables are independent. If $\theta = (\theta_1, \theta_2, \dots, \theta_M)$ is the vector of latent variables, the mean-field assumption implies an approximation of the form,
$$
p(\theta|\mathcal{D}) \approx q_{\phi}(\theta) = \prod_{i=1}^{M} {q_{i}}_{\phi_i}(\theta_i),
$$
where $q_{\phi_i}(\cdot)$ is the approximate marginal posterior over the latent variable $\theta_i$ parameterized by $\phi_i$.

### Evidence Lower Bound (ELBO)

So, to recap, the generic VI strategy is to pose a suitable parameterized family of densities $q_{\phi}(\theta)$ to approximate the true posterior $p(\theta|\mathcal{D})$ and to minimize the KL divergence from $q$ to $p$:
$$
\phi^* = \underset{\phi}{\mathrm{argmin}}\ \mathrm{KL}\left[ q_{\phi}(\theta) || p(\theta|\mathcal{D}) \right].
$$
We cannot actually optimize the KL divergence directly because of it's dependence on the true posterior $p(\theta | \mathcal{D})$.
Instead, we will solve an equivalent, tractable optimization problem. Define the function $\mathcal{L}(\phi)$ as $\mathcal{L}(\phi) =  \mathbb{E}_{q(\theta)}[\log p(\theta, \mathcal{D})] + \mathbb{H}[q(\theta)]$, where, $\mathbb{H}[q(\theta)] =\mathbb{E}_{q(\theta)}[-\log q(\theta)] $  is the *entropy* of $q$. With some simple algebra you can show that solving the optimization problem:
$$
\phi^* = \underset{\phi}{\mathrm{argmax}}\ \mathcal{L}(\phi),
$$
is equivalent to minimizing the KL divergence between $q$ and $p(\theta|\mathcal{D})$.

**Proof**:
$$
\begin{align}
\mathrm{KL}\left[ q_{\phi}(\theta) || p(\theta|\mathcal{D})  \right] &= \mathbb{E}_q \left[  \log \left( \frac{q_{\phi}(\theta)}{p(\theta|\mathcal{D})} \right) \right], \\
&= \mathbb{E}_q \left[ \log \left( \frac{q_{\phi}(\theta) Z}{p(\theta, \mathcal{D})} \right) \right], \ \text{where $Z$ is the evidence,} \\
&= \underset{=-\mathbb{H}[q(\theta)]}{\underbrace{\mathbb{E}_q [\log q_{\phi}(\theta)]}} - \mathbb{E}_q [\log p(\theta, \mathcal{D})] + \underset{\text{this is a constant}}{\underbrace{\log Z}},\\
&= - \mathcal{L}(\phi) + \log Z.
\end{align}
$$
Therefore,
$$
\log Z = \mathrm{KL}\left[ q_{\phi}(\theta) || p(\theta|\mathcal{D})  \right] + \mathcal{L}(\phi).
$$

Now we see that the log evidence (which is a constant) is the sum of the objective function $\mathcal{L}(\phi)$ and the KL divergence between the true and approximate posteriors.
Since, the KL divergence is non-negative, the objective $\mathcal{L}(\phi)$ is a lower-bound on the log evidence. The bound is tight, i.e., $\log Z = \mathcal{L}(\phi)$, if $q_{\phi}$ matches the true posterior perfectly. Minimizing the KL divergence wrt to the variational parameters, $\phi$ is equivalent to maximizing the objective $\mathcal{L}(\phi)$ wrt to $\phi$ and since  $\mathcal{L}(\phi)$ depends on terms that we know and can compute and/or approximate, we use it as the objective function for our VI optimization problem. $\mathcal{L}(\phi)$ is also known as the Evidence Lower Bound or ELBO.

One of the nice things about the ELBO is that it has a neat interpretation. The ELBO is a sum of two terms:
1. $\mathbb{E}_{q(\theta)}[\log p(\theta, \mathcal{D})]$ is a measure of the expected model fit under the approximate posterior density.
2. $\mathbb{H}[q(\theta)]$ - the entropy of the approximate posterior acts a regularizer. The entropy of a distribution is a measure of how "diffuse" it is. In maximizing the entropy, we try to construct our posterior approximation such that it accounts for the maximum ammount of uncertainty in the latent variables conditional on the observed data.

The two terms in the objective function $\mathcal{L}(\phi)$ therefore have an associated trade-off - in optimizing the ELBO we are simultaneously trying to achieve the best possible fit to the data without introducing any excess bias that is not supported by the data (see [the principle of maximum entropy](https://en.wikipedia.org/wiki/Principle_of_maximum_entropy) for assigning probability distributions).

Another nice by-product of doing Bayesian inference by maximizing the ELBO is that we can perform Bayesian model selection. Bayesian model selection relies on the estimation and comparison of the model evidence $Z$ (or it's log) and in VI we work with an approximation to this quantity.

### Automatic Differentiation Variational Inference (ADVI)

Suppose you have put together the joint probability model $p(\theta, \mathcal{D})$. The latent variables that have to be inferred are $\theta = (\theta_1, \theta_2, \dots, \theta_M)$. Variational inference in generic probability models can become extremely tedious and complicated due to the fact the individual $\theta_i$s may come from different probability spaces and have different supports. This means that the user must pose appropriate variational distributions for each $\theta_i$ and derive gradients of the probability model, $p$, wrt to the individual latent variables separately. Furthermore, taking the gradient of the ELBO wrt to the variational parameters require differentiating through a sampling procedure for approximating the datafit term - $\mathbb{E}_{q(\theta)}[\log p(\theta, \mathcal{D})] \approx \frac{1}{S} \sum_{s=1}^{S}\log p(\theta^{(s)}, \mathcal{D}), \theta^{(s)} \sim q(\theta)$.
It turns out that the estimator of ELBO gradient obtained this way has very high variance.
This high variance problem is alleviated by means of the *reparameterization trick* (see Kingma's paper on Autoencoding Variational Bayes).

To the greatest extent possible, we would like to automate the variational inference procedure and for this we will explore the ADVI approach to variational inference. ADVI requires the user to specify two things only -
1. the joint probability model $p(\theta, \mathcal{D})$, and,
2. the dataset $\mathcal{D}$.

**How does ADVI work?**

1. First, ADVI transforms all latent variables, i.e. all $\theta_i$s into new variables $\zeta_i$s by means of a suitable invertible transformation, i.e.,  $\zeta_i = \mathcal{T}(\theta_i)$ such that $\zeta_i$ will have support on the entire real space (recall from our discussion on MCMC with `PyMC3` that this transformation happened by default when specifying  `PyMC3` probability models).

2. Now that all latent variables have same support, ADVI proceeds to specify a common family of distributions on all latent variables. The usual choice is to specify a multivariate Gaussian approximation:
$$
q_{\phi}(\theta) = \mathrm{MVN}(\theta| \mu , \Sigma),
$$
where, $\phi = \{ \mu, \Sigma \}$ denotes the variational parameters.

3. The approximate posterior is further reparameterized in terms of a standard Gaussian to remove the dependence of the sampling procedure from $\phi$.

4. Use standard stochastic optimization techniques to obtain estimates of the variational parameters.

## Example 1 - normal-normal model

Let's demonstrate the VI process end-to-end with a simple example.

Consider the task of inferring the gravitational constant from data (seen previously in Lec. 6).

We perform an experiment $X_n$ that measures the acceleration of gravity and that we know that the measurement variance is $\sigma = 0.1$.

```python
import scipy.constants
g_true = scipy.constants.g
# Generate some synthetic data
N = 10
sigma = 0.2
data = g_true + sigma * np.random.randn(N)
plt.plot(np.arange(N), data, 'o', label='Data')
plt.plot(np.linspace(0, N, 100), g_true*np.ones(100), '--', label='True value')
plt.legend(loc='best');
```

So, we have:
$$
X_n | g, \sigma \sim N(g, \sigma^2).
$$
So, the model says that the measured acceleration of gravity is around the true one with some Gaussian noise.

Assume that our prior state-of-knowledge over $g$ is:
$$
g | g_0, s_0 \sim N(g_0, s_0^2),
$$
with known $g_0 = 10$, $s_0 = 0.4$.

This is a conjugate model and the posterior over $g$ is given by:
$$
g|X \sim N(\tilde{g}, \tilde{s}^2),
$$
where, $\tilde{s}^2 = \left( \frac{N}{\sigma^2} + \frac{1}{s_0^2} \right)^{-1}$ and $\tilde{g} = \tilde{s}^2 \left( \frac{g_0}{s_0^2} + \frac{\sum_{i=1}^{N} X_i}{\sigma^2}\right)$.

```python
def post_mean_and_variance(prior_mean, prior_variance, data, likelihood_var):
    N = len(data)
    sigma2 = likelihood_var
    s02 = prior_variance
    m0 = prior_mean
    sumdata = np.sum(data)
    post_prec = (N/sigma2) + (1./s02)
    post_var = 1./post_prec
    post_mean = post_var * ((m0/s02) + (sumdata/sigma2))
    return post_mean, post_var

gtilde, s2tilde = post_mean_and_variance(10., 0.4**2, data, 0.1**2)

xs1 = np.linspace(7, 12, 100)
xs2 = np.linspace(8, 11, 100)
plt.plot(xs1, st.norm(loc=10., scale=0.4).pdf(xs1), label='Prior')
plt.plot(xs2, st.norm(loc=gtilde, scale=np.sqrt(s2tilde)).pdf(xs2), label='Posterior')
plt.legend(loc='best');
```

Now let's try to infer the posterior over $g$ using VI. Let specify our joint log probability model first.

```python
from autograd.scipy.stats.norm import logpdf as normlogpdf
from autograd import grad, elementwise_grad as egrad
from autograd import numpy as anp
from autograd.numpy import random as npr
from autograd.misc.optimizers import adam

g0, s0 = 10., 0.4
sigma = 0.1

def logprior(g):
    return normlogpdf(g, g0, s0)

def loglikelihood(g):
    return anp.sum(normlogpdf(data, g, sigma))

def logjoint(g):
    return logprior(g) + loglikelihood(g)
```

Now we need to specify a parameterized approximate posterior, $q_{\phi}(\cdot)$. The obvious choice here is a Gaussian:
$$
q_{\phi}(g) = N(g | \phi_1, \exp(\phi_2)^2),
$$
where, $\phi = (\phi_1, \phi_2)$ are the variational parameters. The ELBO needs to be maximized wrt to $\phi$. Let go ahead and set up the ELBO. Recall that the ELBO is given by:
$$
\mathcal{L}(\phi) =  \mathbb{E}_{q(\theta)}[\log p(\theta, \mathcal{D})] + \mathbb{H}[q(\theta)].
$$

To optimize the ELBO, we will need to compute an expectation over the variational distribution $q$ (first term on the RHS in the above equation). This cannot be done analytically. Instead, we resort to a Monte Carlo approximation:

$$
\mathbb{E}_q [\log p(\theta, \mathbf{x})] \approx \frac{1}{S}\sum_{s=1}^{S}   \log p(\theta^{(s)}, \mathbf{x}),
$$
where the samples $\theta^{(s)}$ are drawn from $q$.

```python
def norm_entropy(s):
    s2 = s*s
    return 0.5 * (1. + anp.log(2.*anp.pi*s2))

def ELBO(phi, num_samples):
    m, s = phi[0], anp.exp(phi[1])

    # compute the entropy
    entropy = norm_entropy(s)

    # compute the avg. datafit
    samples = m + s*npr.randn(num_samples)
    datafit = 0.
    for sample in samples:
        datafit += logjoint(sample)
    datafit = datafit/num_samples

    # return the elbo
    elbo = datafit + entropy
    return elbo

def negELBO(phi, num_samples=5):
    return -ELBO(phi, num_samples)
```

Now let's optimize the lower bound using simple stochastic gradient descent (SGD).

```python
# get the gradient of the negative elbo
gradnegelbo = grad(negELBO)

# give initial values to the variational parameters
phi_init = anp.array([9., -1.])

# optimize
num_iter = 1000
phi_i = phi_init
step_size = 1e-3
num_samples = 10
elbos = []
bestnegelbo = np.inf
for i in range(num_iter):
    negelbo = negELBO(phi_i, num_samples)
    elbos.append(-negelbo)
    grad_i = gradnegelbo(phi_i, num_samples)
    phi_next = phi_i - step_size*grad_i

    if (i+1)%50 == 0:
        print('Iteration [%4d] : ELBO = %.5f'%(i+1, -negelbo))
        if negelbo < bestnegelbo:
            phi_opt = phi_i
            bestnegelbo = negelbo
    phi_i = phi_next
```

    Iteration [  50] : ELBO = -9.21201
    Iteration [ 100] : ELBO = -4.84956
    Iteration [ 150] : ELBO = -4.09946
    Iteration [ 200] : ELBO = -5.56783
    Iteration [ 250] : ELBO = -4.04894
    Iteration [ 300] : ELBO = -3.92402
    Iteration [ 350] : ELBO = -3.97946
    Iteration [ 400] : ELBO = -5.41017
    Iteration [ 450] : ELBO = -3.71515
    Iteration [ 500] : ELBO = -4.18543
    Iteration [ 550] : ELBO = -4.49978
    Iteration [ 600] : ELBO = -3.88569
    Iteration [ 650] : ELBO = -3.79997
    Iteration [ 700] : ELBO = -4.83628
    Iteration [ 750] : ELBO = -4.34290
    Iteration [ 800] : ELBO = -4.38020
    Iteration [ 850] : ELBO = -3.61004
    Iteration [ 900] : ELBO = -4.05194
    Iteration [ 950] : ELBO = -3.96508
    Iteration [1000] : ELBO = -4.10269

```python
plt.figure(figsize=(8, 4))
plt.plot(np.arange(num_iter), elbos)
plt.title('ELBO vs SGD Iterations');
```

```python
# get the posterior distribution
postmean = phi_opt[0]
poststdev = np.exp(phi_opt[1])
gpost = st.norm(postmean, poststdev)
```

```python
# compare the VI posterior with the analytical distribution
xs = np.linspace(9, 10.1, 100)
plt.figure(figsize=(10, 6))
plt.plot(xs, st.norm(loc=gtilde, scale=np.sqrt(s2tilde)).pdf(xs), label='True Posterior', linewidth=2.5)
plt.plot(xs, gpost.pdf(xs), label='VI Posterior', linewidth=2.5)
plt.legend(loc='best', fontsize=20);
```

As you can see our approximation of the posterior is not exact, but close. The normal-normal model is a very simple example with 1 latent variable.  In practice setting up the variational posterior for all latent variables, keeping track of transformations and optimizing the variational parameters can become highly tedious for models of any reasonable level of complexity. From this point on, we will use `PyMC3`'s ADVI capabilities.

## Example 2 - Coin-toss example

Just like in the MCMC lecture, let's look at the process of setting up a model and performing variational inference and diagnostics with the coin toss example.

The probabilistic model is as follows. We observe binary coin toss data:
$$
x_i|\theta \overset{\mathrm{i.i.d.}}{\sim} \mathrm{Bernoulli}(\theta),
$$
for $i=1, \dots, N$.

The prior over the latent variable $\theta$ is a Beta distribution:
$$
\theta \sim \mathrm{Beta}([2, 2]).
$$
We assign the prior as a Beta distribution with shape parameters 2 and 2, corresponding to a weak apriori belief that the coin is most likely fair.

```python
thetaprior = st.beta(2., 2.)
x = np.linspace(0.001, 0.999, 1000)
plt.plot(x, thetaprior.pdf(x), linewidth=2.5)
plt.title('Prior');
```

We wish to perform posterior inference on $\theta$:
$$
p(\theta| x_1, \dots, x_N) \propto p(\theta) \prod_{i=1}^{N} p(x_i | \theta).
$$
Since this is a conjugate model, we know the posterior in closed form:
$$
\theta | x_1, \dots, x_N = \mathrm{Beta}(\theta, 2+ \sum_{i=1}^N x_i, 2 + N - \sum_{i=1}^Nx_i )
$$

Let's generate some fake data and get the analytical posterior for comparison.

```python
thetatrue =0.3
N = 25
data = np.random.binomial(1, thetatrue, size=(N,))
nheads = data.sum()
ntails = N - nheads
theta_post = st.beta(2. + nheads, 2. + ntails)

# plot data
plt.figure(figsize=(12, 5))
plt.subplot(121)
_=plt.bar(*np.unique(data, return_counts=True), width=0.2)
_=plt.xticks([0, 1], fontsize=15)
_=plt.title('Observed H/T frequencies', fontsize = 15)

# plot posterior
plt.subplot(122)
x = np.linspace(0.001, 0.999, 1000)
postpdf = theta_post.pdf(x)
y = np.linspace(0., np.max(postpdf), 100)
plt.plot(x, postpdf, linewidth=2.5, label='Posterior')
plt.plot(x, thetaprior.pdf(x), linewidth=2.5, label='Prior')
plt.plot(thetatrue*np.ones_like(y), y, linewidth=2.5, linestyle='--', label='True $\\theta$')
plt.legend(loc='best', fontsize=15)
plt.xticks(fontsize=15)
plt.title('Coin Toss Bayesian Inference', fontsize = 15);

plt.tight_layout()
```

Now let's setup the `PyMC3` model and do variational inference. To do this, we simply need to setup the model like in the MCMC setup, and call `pymc3.variational.ADVI` to do inference.

```python
model = pm.Model()
with model:
    theta = pm.Beta('theta', 2., 2.)
    x = pm.Bernoulli('x', theta, observed=data)
model
```

$$
            \begin{array}{rcl}
            \text{theta} &\sim & \text{Beta}(\mathit{alpha}=2.0,~\mathit{beta}=2.0)\\\text{x} &\sim & \text{Bernoulli}(\mathit{p}=\text{theta})
            \end{array}
            $$

```python
num_iter = 20000
num_samples = 20
with model:
    method = pm.ADVI()  # <- This method makes q a diagonal Gaussian
    #method = pm.FullRankADVI() # <- this method makes q a Gaussian with full rank cov. matrix
    #method = pm.SVGD() # <- Stein Variational Gradient descent (see additional readings)
    #method = pm.NFVI() # <- Normalizing flow Variational inference (see additional readings)
    vi_approx = pm.fit(n=num_iter,
                       method=method,
                       progressbar=True,
                       obj_n_mc=num_samples)
```

    Average Loss = 16.902: 100%|██████████| 20000/20000 [00:14<00:00, 1359.17it/s]
    Finished [100%]: Average Loss = 16.902

```python
# Let's check the optimization convergence
plt.figure(figsize=(10, 6))
plt.plot(np.arange(num_iter), -vi_approx.hist)
plt.ylabel('ELBO', fontsize=15)
plt.xlabel('Iteration', fontsize=15)
plt.yticks(fontsize=15)
plt.xticks(fontsize=15);
```

Clearly, we see that the optimization has converged.

Recall that ADVI transforms every variable with finite support with an invertible function to obtain a variable with support over the entire $\mathbb{R}$. The variational posterior is then set to be a diagonal Gaussian.
In the coin toss problem, the latent variable $\theta \in (0, 1)$ is transformed with the logit function $g(\theta ) = \log \frac{\theta}{1 - \theta}$, i.e., `PyMC3` works with the transformed variable $\tilde{\theta} = g(\theta)$.
The posterior over $\tilde{\theta}$ is then approximated with a Gaussian $q_{\phi}(\tilde{\theta}) = \mathcal{N}(\mu, \sigma^2)$, where, $\phi = (\mu, \sigma)$ are the variational parameters. Let's get these parameters and visualize the posterior.

```python
# get the variational parameters a
mu, sigma = vi_approx.mean.eval(), vi_approx.std.eval()
qtheta_transformed = pm.Normal.dist(mu, sigma)
transform = pm.transforms.logodds
def qthetalogp(x):
    x = tt.as_tensor(x)
    y = transform.forward(x)
    logpy = qtheta_transformed.logp(y)
    logpx = logpy + transform.jacobian_det(x)
    return logpx.eval()
```

```python
x = np.linspace(0.001, 0.999, 1000)
postpdf = theta_post.pdf(x)
postpdf_vi = np.exp(qthetalogp(x))
y = np.linspace(0., np.max(postpdf_vi), 100)
plt.plot(x, postpdf_vi, linewidth=2.5, label='Posterior (ADVI)')
plt.plot(thetatrue*np.ones_like(y), y, linewidth=2.5, linestyle='--', label='True $\\theta$')
plt.legend(loc='best', fontsize=15)
plt.xticks(fontsize=15)
plt.title('Coin Toss Variational Bayesian Inference', fontsize = 15);
```

###  Posterior predictive distribution

To estimate posterior predictive expectations, we first create a `MultiTrace` object out of the results of the ADVI stored in `vi_approx`.

```python
trace = vi_approx.sample(2000)
```

From this point all of the computation is exactly as it was when doing MCMC. Let's generate some synthetic datasets from the posterior predictive distribution.

```python
pp_samples = pm.sample_posterior_predictive(trace=trace, samples=500, model=model)
x_post = pp_samples['x']
x_post.shape # num samples of theta \times size of the dataset
```

    /Users/rohitkt10/anaconda3/lib/python3.6/site-packages/pymc3/sampling.py:1247: UserWarning: samples parameter is smaller than nchains times ndraws, some draws and/or chains may not be represented in the returned posterior predictive sample
      "samples parameter is smaller than nchains times ndraws, some draws "
    100%|██████████| 500/500 [00:00<00:00, 2854.66it/s]

    (500, 25)

```python
plt.figure( figsize=(10, 10) )
for i in range(20):
    plt.subplot(5, 4, i+1)
    plt.bar(*np.unique(x_post[i], return_counts=True), width=0.2)
    plt.bar(*np.unique(data, return_counts=True), width=0.12,
            alpha=0.5, label='Observed data')
    plt.xticks([0, 1])
    plt.legend(loc='best', fontsize=8)
```

## Example 3 - Challenger Space Shuttle Disaster

Let's revisit this example from the MCMC lecture.

```python
# load data
challenger_data = np.genfromtxt("challenger_data.csv", skip_header=1,
                                usecols=[1, 2], missing_values="NA",
                                delimiter=",")
challenger_data = challenger_data[~np.isnan(challenger_data[:, 1])]
print("Temp (F), O-Ring failure?")
print(challenger_data)
```

    Temp (F), O-Ring failure?
    [[66.  0.]
     [70.  1.]
     [69.  0.]
     [68.  0.]
     [67.  0.]
     [72.  0.]
     [73.  0.]
     [70.  0.]
     [57.  1.]
     [63.  1.]
     [70.  1.]
     [78.  0.]
     [67.  0.]
     [53.  1.]
     [67.  0.]
     [75.  0.]
     [70.  0.]
     [81.  0.]
     [76.  0.]
     [79.  0.]
     [75.  1.]
     [76.  0.]
     [58.  1.]]

```python
# plot it, as a function of temperature (the first column)
plt.figure(figsize=(12, 5))
plt.plot(challenger_data[:, 0], challenger_data[:, 1], 'ro',
         markersize=15)
plt.ylabel("Damage Incident?",fontsize=20)
plt.xlabel("Outside temperature (Fahrenheit)",fontsize=20)
plt.title("Defects of the Space Shuttle O-Rings vs temperature",
          fontsize=20)
plt.yticks([0, 1], fontsize=15)
plt.xticks(fontsize=15)
plt.tight_layout()
```

### Probabilistic model

The defect probability is modeled as a function of the outside temperature:
$$\sigma(t) = \frac{1}{ 1 + e^{ \;\beta t + \alpha } } $$.

The goal is to infer the latent variables $\alpha$ and $\beta$.

We set normal priors on the latent variables - $\alpha = \mathcal{N}(0, 10^2)$ and $\beta = \mathcal{N}(0, 10^2)$ and the likelihood model is giveb by $p(x_i | \alpha, \beta, t) = \mathrm{Bern}(x_i | \sigma(t; \alpha, \beta) )$.

The graphical model for this problem is shown below.

```python
gcp = Digraph('space_shuttle_disaster')

# setup the nodes
gcp.node('alpha', label='<α>')
gcp.node('beta', label='<β>')
with gcp.subgraph(name='cluster_0') as sg:
    sg.node('pi', label='<p<sub>i</sub>>')
    sg.node('xi', label='<x<sub>i</sub>>', style='filled')
    sg.attr(color='blue')
    sg.attr(label='i=1,2...')
    sg.attr(labelloc='b')

# setup the edges
gcp.edge('alpha', 'pi')
gcp.edge('beta', 'pi')
gcp.edge('pi', 'xi')
gcp
```

###  `PyMC3` model

```python
# gather the data and apply preprocessing if any
temp = challenger_data[:, 0]
temp_scaled = (temp - np.mean(temp))/np.std(temp)
data = challenger_data[:, 1]

# instantiate the pymc3 model
challenger_model = pm.Model()

# define the graph
with challenger_model:
    # define the prior
    alpha = pm.Normal('alpha', mu=0., sigma=10.)
    beta = pm.Normal('beta', mu=0., sigma=10.)

    # get the probabilities of failure at each observed temp
    p = pm.Deterministic('p', 1./(1. + tt.exp(alpha + beta*temp_scaled)))

    # define the likelihood
    x = pm.Bernoulli('x', p=p, observed=data)
print("Challenger space shuttle disaster model:")
challenger_model
```

    Challenger space shuttle disaster model:

$$
            \begin{array}{rcl}
            \text{alpha} &\sim & \text{Normal}(\mathit{mu}=0.0,~\mathit{sigma}=10.0)\\\text{beta} &\sim & \text{Normal}(\mathit{mu}=0.0,~\mathit{sigma}=10.0)\\\text{p} &\sim & \text{Deterministic}(\text{Constant},~\text{Constant},~\text{alpha},~\text{beta},~\text{Constant})\\\text{x} &\sim & \text{Bernoulli}(\mathit{p}=\text{p})
            \end{array}
            $$

### Inference and posterior visualization

Now let's infer the hidden parameters with VI. We will use both mean-field (i.e. diagonal covariance Gaussian) ADVI and full-rank (i.e. full rank covariance matrix Gaussian) ADVI.

```python
num_iter = 20000
num_samples = 20
with challenger_model:
    method_1 = pm.ADVI()  # <- This method makes q a diagonal Gaussian
    vi_approx_1 = pm.fit(n=num_iter,
                       method=method_1,
                       progressbar=True,
                       obj_n_mc=num_samples)
```

    Average Loss = 15.671: 100%|██████████| 20000/20000 [00:15<00:00, 1327.99it/s]
    Finished [100%]: Average Loss = 15.672

```python
with challenger_model:
    method_2 = pm.FullRankADVI()  # <- This method makes q a diagonal Gaussian
    vi_approx_2 = pm.fit(n=num_iter,
                       method=method_2,
                       progressbar=True,
                       obj_n_mc=num_samples)
```

      0%|          | 0/20000 [00:00<?, ?it/s]/Users/rohitkt10/anaconda3/lib/python3.6/site-packages/theano/tensor/subtensor.py:2339: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
      out[0][inputs[2:]] = inputs[1]
    /Users/rohitkt10/anaconda3/lib/python3.6/site-packages/theano/tensor/basic.py:6611: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
      result[diagonal_slice] = x
    /Users/rohitkt10/anaconda3/lib/python3.6/site-packages/theano/tensor/subtensor.py:2197: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
      rval = inputs[0].__getitem__(inputs[1:])
    Average Loss = 15.631: 100%|██████████| 20000/20000 [00:19<00:00, 1017.13it/s]
    Finished [100%]: Average Loss = 15.63

Let's make sure that the ELBO has converged.

```python
plt.figure(figsize=(15, 15))

# ADVI
plt.subplot(211)
plt.plot(np.arange(num_iter), -vi_approx_1.hist)
plt.ylabel('ELBO', fontsize=15)
plt.xlabel('Iteration', fontsize=15)
plt.yticks(fontsize=15)
plt.xticks(fontsize=15)
plt.title('ADVI', fontsize=15);

# Full rank ADVI
plt.subplot(212)
plt.plot(np.arange(num_iter), -vi_approx_2.hist)
plt.ylabel('ELBO', fontsize=15)
plt.xlabel('Iteration', fontsize=15)
plt.yticks(fontsize=15)
plt.xticks(fontsize=15)
plt.title('Full-rank ADVI', fontsize=15)

plt.tight_layout()
```

We see that both methods have converged to a local minimum.

Let's get the variational posteriors $q$ for each case and compare.

```python
# get the variational parameters and compute the post. pdf over a grid for ADVI
mu, sigma = vi_approx_1.mean.eval(), vi_approx_1.std.eval()
qtheta = pm.MvNormal.dist(mu=mu, cov=np.diag(sigma), shape=2)
x1max, x2max = mu + 3.*sigma
x1min, x2min = mu - 3.*sigma
x1 = np.linspace(x1min, x1max, 50)
x2 = np.linspace(x2min, x2max, 50)
names = [g.name for g in vi_approx_1.groups[0].group]
X1, X2 = np.meshgrid(x1, x2)
Xgrid = np.vstack([X1.flatten(), X2.flatten()]).T
advipdf = np.exp(qtheta.logp(tt.as_tensor(Xgrid)).eval()).reshape((50, 50))
```

```python
# get the variational parameters and compute the post. pdf over a grid for full rank ADVI
mu, cov = vi_approx_2.mean.eval(), vi_approx_2.cov.eval()
sigma = np.sqrt(np.diag(cov))
qtheta = pm.MvNormal.dist(mu=mu, cov=cov, shape=2)

x1max, x2max = mu + 3.*sigma
x1min, x2min = mu - 3.*sigma
x1 = np.linspace(x1min, x1max, 50)
x2 = np.linspace(x2min, x2max, 50)
names = [g.name for g in vi_approx_2.groups[0].group]
X1, X2 = np.meshgrid(x1, x2)
Xgrid = np.vstack([X1.flatten(), X2.flatten()]).T
advifullrankpdf = np.exp(qtheta.logp(tt.as_tensor(Xgrid)).eval()).reshape((50, 50))
```

    /Users/rohitkt10/anaconda3/lib/python3.6/site-packages/theano/tensor/subtensor.py:2339: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
      out[0][inputs[2:]] = inputs[1]

```python
plt.figure(figsize=(15, 6))

plt.subplot(121)
plt.contourf(X1, X2, advipdf, 50)
plt.xlabel('$\\'+names[0]+'$', fontsize=15)
plt.ylabel('$\\'+names[1]+'$', fontsize=15)
plt.xticks(fontsize=15)
plt.yticks(fontsize=15)
plt.title('Joint Posterior over $(\\alpha, \\beta)$ - diagonal Gaussian posterior.', fontsize=15)

plt.subplot(122)
plt.contourf(X1, X2, advifullrankpdf, 50)
plt.xlabel('$\\'+names[0]+'$', fontsize=15)
plt.ylabel('$\\'+names[1]+'$', fontsize=15)
plt.xticks(fontsize=15)
plt.yticks(fontsize=15)
plt.title('Joint Posterior over $(\\alpha, \\beta)$ - full rank covariance.', fontsize=15)

plt.tight_layout()
```

Note how the simple diagonal Gaussian does not capture correlation between $\alpha$ and $\beta$,something we are able to recover with the full rank ADVI.

###  Posterior predictive distribution

Here we will use a trace generated from the full rank Gaussian posterior to make posterior predictions.

```python
trace = vi_approx_2.sample(10000)

with challenger_model:
    ppsamples = pm.sample_posterior_predictive(trace=trace,
                                               samples=2000,
                                               var_names=['p'])['p']
```

    /Users/rohitkt10/anaconda3/lib/python3.6/site-packages/theano/tensor/subtensor.py:2339: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
      out[0][inputs[2:]] = inputs[1]
    /Users/rohitkt10/anaconda3/lib/python3.6/site-packages/pymc3/sampling.py:1247: UserWarning: samples parameter is smaller than nchains times ndraws, some draws and/or chains may not be represented in the returned posterior predictive sample
      "samples parameter is smaller than nchains times ndraws, some draws "
    100%|██████████| 2000/2000 [00:00<00:00, 4036.76it/s]

```python
# get posterior predictive mean and 95% interval
ppmean = ppsamples.mean(axis=0)
pp_lower, pp_upper = np.percentile(ppsamples, axis=0, q=[2.5, 97.5])
```

```python
plt.figure(figsize=(15, 8))
plt.plot(temp, data, 'ro', markersize=12, label='Observed data')
idx=np.argsort(temp)
plt.plot(temp[idx], ppmean[idx], linestyle='--', linewidth=2.5,
         label='Post. pred. mean prob.')
plt.fill_between(temp[idx], pp_lower[idx], pp_upper[idx],
                color='purple', alpha=0.25, label='95% Confidence')
plt.ylabel("Probability estimate",fontsize=20)
plt.xlabel("Outside temperature (Fahrenheit)",fontsize=20)
plt.title("Defects of the Space Shuttle O-Rings vs temperature",
          fontsize=20)
plt.yticks(np.arange(0., 1.01, 0.2), fontsize=20)
plt.xticks(fontsize=20)
plt.legend(loc='best', fontsize=20)
plt.tight_layout()
```

```python

```


# PRAKTIKUM 7
__Interpolasi__

*Topik*
1. Interpolasi menggunakan SPL
2. Interpolasi Lagrange
3. Interpolasi Newton

Pada subbab sebelumnya, telah dipelajari bahwa deret Taylor dapat digunakan untuk menghampiri fungsi $ f(x) $. Informasi yang dibutuhkan untuk menyusun fungsi hampiran polinomial Taylor adalah nilai dari $ f $ dan turunannya pada $ x_0 $. Kekurangan metode ini adalah turunan ordo-tinggi harus diketahui. Namun, sering kali turunan tersebut tak terdefinisi ataupun sulit untuk dihitung. Dengan demikian, diperlukan metode lain untuk menghampiri fungsi $ f(x) $ dengan fungsi polinomial tanpa perlu menggunakan turunan.

Misalkan, diketahui fungsi $ y=f(x) $ dengan $ N+1 $ titik, yaitu $ (x_0,y_0),\dots,(x_N,y_N) $, dimana nilai $ x_k $ menyebar pada interval $ [a,b] $ dan memenuhi
\begin{align*}
\begin{array}{ccc}
a\le x_0<x_1<\dots<x_N\le b & \text{dan} & y_k=f(x_k)
\end{array}
\end{align*}
Fungsi polinomial $ P(x) $ dengan derajat $ N $ dapat disusun melewati $ N+1 $ titik tersebut. Metode ini hanya memerlukan nilai numerik dari $ x_k $ dan $ y_k $. Oleh karena itu, turunan fungsi ordo-tinggi tidak lagi diperlukan. Fungsi polinomial $ P(x) $ dapat digunakan untuk menghampiri fungsi $ f(x) $ pada interval $ [a,b] $.

Pada sudut pandang lain, jika fungsi $ f(x) $ hanya diketahui berupa pasangan titik terurut $ (x_k,y_k) $, maka diperlukan metode selain deret Taylor untuk menghampiri fungsi $ f(x) $ berupa persamaan garis. Jika terdapat galat yang signifikan pada data titik terurut, maka metode yang digunakan untuk memperoleh fungsi hampiran adalah percocokan kurva yang akan dibahas pada Bab 8. Sebaliknya, jika diketahui titik $ (x_k,y_k) $ memiliki akurasi yang tinggi, maka fungsi polinomial $ y=P(x) $ harus melewati semua titik yang diketahui. Jika nilai $ x_0\le x\le x_N $ maka hampiran $ P(x) $ disebut sebagai nilai \textbf{interpolasi}. Sementara itu, jika nilai $ x\le x_0 $ atau $ x>x_N $ maka hampiran $ P(x) $ disebut sebagai nilai \textbf{ekstrapolasi}. Hampiran polinomial digunakan untuk merancang algoritma \textit{software} sebagai hampiran suatu fungsi, untuk turunan dan integral numerik, dan membuat kurva yang melewati beberapa titik. Metode paling dasar yang dapat digunakan untuk membentuk persamaan garis interpolasi adalah menggunakan sistem persamaan linear.

# 1 Interpolasi Linear

### Contoh 1
Carilah koefisien polinomial $ P(x)=A+Bx+Cx^2+Dx^3 $ yang melewati empat titik, yaitu $ (1,1.06) $, $ (2,1.12) $, $ (3,1.34) $, dan $ (5,1.78) $. Gunakan hampiran polinomial yang diperoleh untuk menghitung $ P(4) $ dan $ P(5.5) $.

```julia
using Plots
```

```julia
xk = [1, 2, 3, 5]
yk = [1.06, 1.12, 1.34, 1.78]
scatter(xk,yk,legend = :false)
```

Akan dicari nilai A, B, C, dan D. polinomial $ P(x)=A+Bx+Cx^2+Dx^3 $

Berdasarkan titik-titik yang diketahui, didapatkan persamaan linear untuk setiap $ x=1,2,3,5 $ seperti berikut.
\begin{align*}
\begin{array}{lllrlrlrl}
\text{Ketika } x=1 :& A&+&1B&+&1C &+&1D 	&= 1.06\\
\text{Ketika } x=2 :& A&+&2B&+&4C &+&8D 	&= 1.12\\
\text{Ketika } x=3 :& A&+&3B&+&9C &+&27D 	&= 1.34\\
\text{Ketika } x=5 :& A&+&5B&+&25C&+&125D 	&= 1.78\\
\end{array}
\end{align*}

```julia
A = [1 1 1 1
     1 2 4 8
     1 3 9 27
     1 5 25 125];
B = [1.06; 1.12; 1.34; 1.78];
C = A\B
```

```julia
x = 1:0.1:5
f(x) = C[1]+C[2]*x+C[3]*x^2+C[4]*x^3
plot!(x,f.(x))
```

```julia
f(4),f(5.5)
```

Jadi, fungsi polinomial yang melewati keempat titik yang diberikan adalah $$ P(x)=1.28-0.4x+0.2x^2-0.02x^3 $$

# 2 Interpolasi Lagrange
Berikut merupakan bentuk umum interpolasi Lagrange.

Hampiran polinomial $ P_N(x) $ dengan derajat paling tinggi $ N $ dapat disusun menggunakan $ N+1 $ titik yaitu $ (x_0,y_0) $, $ (x_1,y_1) $, $ \dots $, $ (x_N,y_N) $ dan memiliki bentuk umum
\begin{align}\label{eq:7 lagrange umum}
P_N(x) = \sum_{k=0}^{N}{y_k L_{N,k}(x)}
\end{align}
dimana $ L_{N,k}(x) $ adalah koefisien polinomial Lagrange, yaitu
\begin{equation*}
L_{N,k}(x)=\dfrac{(x-x_0)\dots(x-x_{k-1})(x-x_{k+1})\dots(x-x_N)}
{(x_k-x_0)\dots(x_k-x_{k-1})(x_k-x_{k+1})\dots(x_k-x_N)}
\end{equation*}

Hampiran polinomial Lagrange kuadratik $ P_2(x) $ dapat disusun menggunakan 3 titik $ (x_0,y_0) $, $ (x_1,y_1) $, dan $ (x_2,y_2) $, yaitu
\begin{align}\label{eq:7 lagrange kuadratik}
P_2(x) = y_0 \dfrac{(x-x_1)(x-x_2)}{(x_0-x_1)(x_0-x_2)} + y_1  \dfrac{(x-x_0)(x-x_2)}{(x_1-x_0)(x_1-x_2)} + y_2  \dfrac{(x-x_0)(x-x_1)}{(x_2-x_0)(x_2-x_1)}
\end{align}

Misal, diberikan fungsi $ f(x)=\cos(x) $ pada interval $ [0,1.2] $.

Gunakan titik $ x_0=0 $, $ x_1=0.6 $, dan $ x_2=1.2 $ untuk menyusun interpolasi polinomial kuadratik $ P_2(x) $.

\begin{align*}
P_2(x) = & 1.00000
\dfrac{(x-0.6)(x-1.2)}{(0-0.6)(0-1.2)} + 0.82534 \dfrac{(x-0)(x-1.2)}{(0.6-0)(0.6-1.2)} + 0.36236 \dfrac{(x-0)(x-0.7)}{(1.2-0)(1.2-0.6)}\\
= & -0.400435x^2  -0.050846x   + 1
\end{align*}

```julia
#= Pseudo-Code
%%METODE INTERPOLASI LAGRANGE
% y = lagrange(x,xd,yd)
% Input  : x     -> x dapat berupa titik tunggal, atau kumpulan titik
%          xd,yd -> Data pasangan terurut (X,Y)
% Output : y     -> hasil interpolasi dari titik x
=#
function lagrange(x, xd, yd)
  #% sedikit trik agar bisa menghitung nilai interpolasi pada banyak
  #% titik sekaligus. Intinya, tetap diproses satu titik demi titik.
  m=length(x);
  y=zeros(m)
  if m > 1
    for i=1:m
      # proses satu titik demi satu titik
      y[i]=lagrange(x[i], xd, yd);
    end
    return y
  end
  #% periksa jumlah titik dan tentukan derajat polinom
  ntitik = length(xd);
  n = ntitik-1;  #% derajat maksimum, sesuai jumlah titik yang ada
  #% hitung L_{n,k}(x) untuk k=1:(n+1)
  Ln=ones(1,ntitik);
  for i=1:ntitik
    for j=1:ntitik
      if i!=j
        Ln[i] = Ln[i] * (x-xd[j])/(xd[i]-xd[j]);;
      end
    end
  end
  #% hitung P_{N}(x)
  y = 0;
  for i=1:ntitik
    y = y + yd[i]*Ln[i];
  end
  return y
end
```

### Contoh 2
Diberikan fungsi $ f(x)=\cos(x) $ pada interval $ [0,1.2] $.

Hitung dan gambarkan hampiran polinomial $ P_2(x) $ dari fungsi $ f $ menggunakan interpolasi Lagrange beserta galat dari masing-masing hampiran pada interval [0,1.5].

```julia
f(x) = cos(x);
xd = [0,0.6,1.2];
yd = f.(xd);
x = 0:0.01:1.5
p2 = lagrange(x,xd,yd)

plt1 = plot(x,f.(x),label = "cos(x)")
plot!(x,p2,label = "interpolasi lagrange")
scatter!(xd,yd,label = :none)
xlabel!("x")
ylabel!("y")
```

```julia
f(x) = cos(x);
xd = [0,0.4,0.8,1.2];
yd = f.(xd);
x = 0:0.01:1.5
p3 = lagrange(x,xd,yd)

plt2 = plot(x,f.(x),label = "cos(x)")
plot!(x,p3,label = "interpolasi lagrange")
scatter!(xd,yd,label = :none)
xlabel!("x")
ylabel!("y")
```

```julia
# Gunakan fungsi f(x), nilai x dan p2 untuk membentuk plot galat
# hitung nilai galat masing-masing hampiran interpolasi
e2 = abs.(f.(x)-p2);
# Plot galat E2(x)
plt3 = plot(x,e2,label = "E2(x)")
xlabel!("x")
ylabel!("y")
```

```julia
e3 = abs.(f.(x)-p3);
# Plot galat E3(x)
plt4 = plot(x,e3,label = "E3(x)")
xlabel!("x")
ylabel!("y")
```

# 3 Interpolasi Newton
Jika interpolasi Lagrange digunakan, maka diperlukan waktu komputasi yang cukup lama karena tidak terdapat hubungan antara $ P_{N-1}(x) $ dan $ P_N(x) $.

Pendekatan lain yang dapat digunakan untuk membentuk hampiran polinomial adalah menggunakan interpolasi Newton. Interpolasi Newton memiliki pola rekursif yaitu
\begin{align}\label{eq:7 Newton 1}
\begin{split}
P_1(x)=&\ a_0+a_1(x-x_0)\\
P_2(x)=&\ a_0+a_1(x-x_0)+a_2(x-x_0)(x-x_1)\\
P_3(x)=&\ a_0+a_1(x-x_0)+a_2(x-x_0)(x-x_1)+a_3(x-x_0)(x-x_1)(x-x_2)\\
&\ \ \ \vdots\\
P_N(x)=&\ a_0+a_1(x-x_0)+a_2(x-x_0)(x-x_1)+a_3(x-x_0)(x-x_1)(x-x_2)\\
&+\dots+a_N(x-x_0)\dots(x-x_{N-1})
\end{split}
\end{align}

Hubungan antara polinomial $ P_{N-1}(x) $ dan $ P_N(x) $ adalah
$ P_N(x)=\ P_{N-1}(x)+a_N(x-x_0)(x-x_1)\dots(x-x_{N-1}) $

Nilai $a_0$, $a_1$, $a_2$, dan seterusnya dapat dicari menggunakan tabel selisih-terbagi atau beda-terbagi

```julia
#= Pseudo-code
%%METODE INTERPOLASI NEWTON
% [y,D] = newton(x,xd,yd)
% Input  : x     -> x dapat berupa titik tunggal, atau kumpulan titik
%          xd,yd -> Data pasangan terurut (X,Y)
% Output : y     -> Hasil interpolasi dari titik x
%          D     -> Tabel beda-terbagi untuk interpolasi Newton
=#

function newton(x, xd, yd)
  #% sedikit trik agar bisa menghitung nilai interpolasi pada banyak
  #% titik sekaligus. Intinya, tetap diproses satu titik demi titik.
  m=length(x);
  y=zeros(m)
  if m > 1
    for i=1:m
      # proses satu titik demi satu titik
      y[i],D =newton(x[i], xd, yd);
    end
    return y,D
  end
  #% periksa jumlah titik dan tentukan derajat polinom
  ntitik = length(xd);
  #% hitung tabel beda-terbagi (divided-difference)
  D = zeros(ntitik,ntitik)
  D[:,1]=yd;          #% kolom pertama
  for j=2:ntitik      #% kolom ke-2 dan seterusnya
    for k=j:ntitik
        D[k,j] = (D[k,j-1]-D[k-1,j-1])/(xd[k]-xd[k-j+1]);
    end
  end
  #% hitung interpolasi Newton
  y = D[1,1]; s = 1;
  for i=2:ntitik
    s = s * (x-xd[i-1]);
    y = y + D[i,i]*s;
  end
  return y, D
end
```

### Contoh 3
Diketahui $ f(x)=x^3-4x $.

Susunlah tabel selisih terbagi berdasarkan nilai $ x_0=1,x_1=2,\dots,x_5=6 $ dan carilah polinomial Newton $ P_3(x) $ berdasarkan $ x_0,x_1,x_2, $ dan $ x_3 $.

Berdasarkan Tabel di atas, didapatkan nilai koefisien $ a_0=-3,a_1=3,a_2=6 $ dan $ a_3=1 $ untuk $ P_3(x) $ pada diagonal tabel selisih terbagi.

Jadi, $ P_3(x) $ adalah
$$ P_3(x)=-3+3(x-1)+6(x-1)(x-2)+(x-1)(x-2)(x-3) $$

```julia
f(x) = x.^3-4*x;
	 xd = 1:6;
	 yd = f(xd);
	 ynew,D = newton(1,xd,yd)
	 D
```

### Contoh 4
Diberikan fungsi $$  f(x)=\cos(x)  $$ dan empat titik $ (k,f(k)) $ dengan $ k=0,1,2,3 $. Berikut merupakan langkah-langkah  untuk membentuk fungsi interpolasi $ P_1(x) $, $ P_2(x) $, dan $ P_3(x) $ menggunakan metode Newton, serta pembentukan grafik perbandingan fungsi $ f(x) $ dan garis interpolasi pada interval $ [0,3.1] $.

```julia
# Definisikan fungsi f(x) dan urutan titik (xd,yd)
f(x) = cos(x);
xd = 0:3;
yd = f.(xd);
# Definisikan interval grafik, yaitu xnew
x = 0:0.1:3.1;
```

```julia
# Hitung nilai ynew dengan interpolasi newton, P1(x)
ynew,D = newton(x,xd[1:2],yd[1:2]);
# Perbandingan plot f(x) dan P1(x)
plt1 = plot(x,f.(x),label = "f(x)")
plot!(x,ynew,label = "P1(x)")
scatter!(xd[1:2],yd[1:2],label = :none)

xlabel!("x")
ylabel!("y")
```

```julia
# Hitung nilai ynew dengan interpolasi newton, P2(x)
ynew,D = newton(x,xd[1:3],yd[1:3]);
# Perbandingan plot f(x) dan P2(x)
plt2 = plot(x,f.(x),label = "f(x)")
plot!(x,ynew,label = "P2(x)")
scatter!(xd[1:3],yd[1:3],label = :none)

xlabel!("x")
ylabel!("y")
```

```julia
# Hitung nilai ynew dengan interpolasi newton, P3(x)
ynew,D = newton(x,xd[1:4],yd[1:4]);
# Perbandingan plot f(x) dan P3(x)
plt3 = plot(x,f.(x),label = "f(x)")
plot!(x,ynew,label = "P3(x)")
scatter!(xd[1:4],yd[1:4],label = :none)

xlabel!("x")
ylabel!("y")
```

### Contoh 5 Soal Kuliah (_Polynomial Wiggle_)
Misalkan diberikan suatu fungsi sebagai berikut.

$$ f(x)=\dfrac{1}{1+12x^2}$$

Dengan mengambil titik $(x_d,y_d)$ pada $x_d=-1,-0.8,-0.6,...,0.6,0.8,1$ dan $y_d=f(x_d)$. Hitunglah
 1. Hampiran interpolasi dari $f(x)$ menggunakan data tersebut.
 2. Hampiran interpolasi dari $f(x)$ dengan membagi data menjadi 2 bagian yaitu $[-1,0]$ dan $[0,1]$

Untuk lebih jelasnya, berikut scatter plot dari data yang ditentukan.

```julia
f(x) = 1/(1+12*x^2);
xd = -1:0.2:1;
yd = f.(xd);
x = -1:0.01:1
scatter(xd,yd,legend = :false)
#plot!(x,f.(x))
```

**Langkah 1** Hampiran interpolasi dari $f(x)$ menggunakan data pada selang $[-1,1]$ secara langsung.

```julia
f(x) = 1/(1+12*x^2);
xd = -1:0.2:1;
yd = f.(xd);
x = -1:.01:1;
ynew = newton(x,xd,yd);

plt = plot(x,f.(x),label = "f(x)")
plot!(x,ynew,label = "P10(x)")
scatter!(xd,yd,label = :none)

xlabel!("x")
ylabel!("y")
```

Terjadi ketidaksesuaian yang amat tinggi pada titik mendekati $x=-1$ dan $x=1$. Ketidaksesuaian itu sering disebut sebagai goncangan polinomial (_polynomial wiggle_). Hal ini terjadi karena proses interpolasi melibatkan fungsi polinomial berderajat terlalu tinggi. Sebagai alternatifnya, derajat fungsi polinomial dapat dipangkas dengan cara mempartisi selang awal menjadi dua atau lebih selang bagian (sub-interval).

**Langkah 2** Hampiran interpolasi dari $f(x)$ dengan membagi data menjadi 2 bagian yaitu $[-1,0]$ dan $[0,1]$

```julia
f(x) = 1/(1+12*x^2);
xd = -1:0.2:0;
yd = f.(xd);
x = -1:.01:0;
ynew = newton(x,xd,yd);

plt = plot(x,f.(x),color = :red,label = "f(x)")
plot!(x,ynew,color = :blue,label = "P5(x)")
scatter!(xd,yd,color = :red,label = :none)

xd = 0:0.2:1;
yd = f.(xd);
x = 0:.01:1;
ynew = newton(x,xd,yd);

plot!(x,f.(x),color = :red,label = :none)
plot!(x,ynew,color = :blue,label = :none)
scatter!(xd,yd,color = :red,label = :none)

xlabel!("x")
ylabel!("y")
```

Hasil interpolasi menggunakan cara kedua jauh lebih akurat daripada menggunakan cara yang pertama

# Soal Latihan
Kerjakan soal berikut pada saat kegiatan praktikum berlangsung.

`Nama: ________`

`NIM: ________`

### Soal 1
Diberikan fungsi $ f(x)=\sin(x) $.

Ulangi langkah-langkah pada **Contoh 2** untuk membentuk hampiran $ P_2(x) $ dan $ P_3(x) $ pada interval $ [0,1.2] $ kemudian hitung galat masing-masing hampiran tersebut.

```julia

```

### Soal 2
Diberikan fungsi $ f(x)=\sin(x) $.
Ulangi **Contoh 4** untuk membentuk hampiran $ P_1(x) $, $ P_2(x) $ dan $ P_3(x) $ dengan titik $ (k,f(k)) $ dimana $ k=0,1,2,3 $ lalu gambarkan masing-masing grafik hampiran Newton dan fungsi $ f(x) $ pada interval $ [0,3.1] $.

```julia

```

### Soal 3
Misalkan diberikan suatu fungsi sebagai berikut.

$$ f(x)=e^{-\sin(x)} $$

Dengan mengambil titik $(x_d,y_d)$ pada $x_d=-2,-1,0,1,2,3,4$ dan $y_d=f(x_d)$. Hitunglah
1. Hampiran interpolasi dari $f(x)$ menggunakan data tersebut.
2. Hampiran interpolasi dari $f(x)$ dengan membagi data menjadi 2 bagian yaitu $[-2,1]$ dan $[1,4]$

```julia

```


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Rounded Nose Cone Radar Cross Section
***

Referring to Section 7.4.1.6, for cones with rounded nose tips, as shown in Figure 7.12, the physical optics approximation with axial incidence is given by (Equation 7.55)

$$
    \sigma = \pi b^2\Bigg[ 1 - \frac{\sin\big(2kb(1-\sin\alpha)\big)}{kb\cos^2\alpha} + \frac{1 + \cos^4\alpha}{4(kb)^2\cos^4\alpha}  - \frac{\cos\big(2kb(1-\sin\alpha)\big)}{2(kb^2)\cos^2\alpha} \Bigg] \hspace{0.5in} \text{(m}^2\text{)},
$$

where $b$ is the radius of the rounded nose tip. For incident angles other than axial, but less than the cone angle ($\theta_i \le \alpha$),  (Equation 7.56)

$$
    \sigma =  \pi b^2 \frac{1 + \theta_i^2}{4(kb)^2} \Big[A_1 + A_2 \cos\big(2k\cos\theta_i (1 - \sin\alpha)\big)  + A_3\sin\big(2k\cos\theta_i(1-\sin\alpha)\big)\Big]\hspace{0.25in} \text{(m}^2\text{)},
$$

where

\begin{align}
    A_1 = \; &2 + 2\alpha^2 - 2\theta_i^2 + \alpha^4 - \alpha^2\theta_i^2 + 0.5\theta_i^4 + 2\alpha^4\theta_i^2 + 4(kb)^2 - 2(kb)^2\theta_i^2 \nonumber \\[5pt]
    &- 8(kb)^3\theta_i^2 + (kb)^2\theta_i^4 + 6(kb)^2\alpha^2\theta_i^2 + 8(kb)^3\theta_i^4 + 13(kb)^4\theta_i^4, \\  \nonumber \\
    A_2 = &-2 -2\alpha^2 + 2\theta_i^2 + \alpha^2\theta_i^2 - 0.5\theta_i^4 - 6(kb)^2\theta_i^2 + 8(kb)^4\theta_i^3 + 3(kb)^2\theta_i^4, \\  \nonumber \\
    A_3 = &-4\big(1 + \alpha^2 - 0.5\theta_i^2 + 3(kb\theta_i)^2\big)\big(kb - kb\theta_i^2 - (kb\theta_i)^2\big) - 4(kb\theta_i)^3.\nonumber
\end{align}
***

Begin by getting the library path

```python
import lib_path
```

Set the operating frequency (Hz), the cone half angle (radians), and the nose radius (m)

```python
from numpy import radians

frequency = 1e9

cone_half_angle = radians(20.0)

nose_radius = 1.4
```

Set the incident angles using the `linspace` routine from `scipy`

```python
from numpy import linspace

incident_angle = linspace(0, cone_half_angle, 1801)
```

Calculate the radar cross section (m^2) for the rounded nose cone

```python
from Libs.rcs.rounded_nose_cone import radar_cross_section

from numpy import array

rcs = array([radar_cross_section(frequency, cone_half_angle, nose_radius, ia) for ia in incident_angle])
```

Display the radar cross section (m^2) for the rounded nose cone using the routines from `matplotlib`

```python
from matplotlib import pyplot as plt

from numpy import log10, degrees

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(degrees(incident_angle), 10.0 * log10(abs(rcs)), '')

# Set the plot title and labels

plt.title('RCS vs Incident Angle', size=14)

plt.ylabel('RCS (dBsm)', size=12)

plt.xlabel('Incident Angle (deg)', size=12)

# Set the tick label size

plt.tick_params(labelsize=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)
```

```python

```


# Grover implementation with 5 qubits and 1 marked state

```python
import numpy as np

# Importing standard Qiskit libraries
from qiskit import QuantumCircuit, transpile, Aer, IBMQ, assemble
from qiskit.tools.jupyter import *
from qiskit.visualization import *
from ibm_quantum_widgets import *
from qiskit.providers.aer import QasmSimulator

# Loading your IBM Quantum account(s)
provider = IBMQ.load_account()
```

    ibmqfactory.load_account:WARNING:2021-09-18 15:49:17,439: Credentials are already in use. The existing account in the session will be replaced.

```python
print("Available backends:")
provider.backends()

```

    Available backends:

    [<IBMQSimulator('ibmq_qasm_simulator') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_armonk') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_santiago') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_bogota') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_belem') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_quito') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_statevector') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_mps') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_extended_stabilizer') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_stabilizer') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_manila') from IBMQ(hub='ibm-q', group='open', project='main')>]

```python
from qiskit.providers.ibmq import least_busy

large_enough_devices = provider.backends(filters=lambda x: x.configuration().n_qubits > 3 and not x.configuration().simulator)
backend = least_busy(large_enough_devices)
print("The best backend is " + backend.name())

```

    The best backend is ibmq_quito

```python
backend = provider.get_backend('ibmq_lima')
```

```python
backend
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>

Here I code Grover's Algorithm with 5 qubits

```python
#def initialization in superposition
def initialization(qc, qubits):
    for qubit in qubits:
        qc.h(qubit)
    return qc
```

```python
example = QuantumCircuit(5)
example_ = initialization(example, range(5))
example_.draw('mpl')
```

```python
#def oracle marking state [0,0,.....1]
# 1 state marked
num_q = 5

from numpy import pi
qc = QuantumCircuit(num_q)

for i in range(num_q -2):
    qc.x(i)
qc.h(num_q -1)
qc.mct(list(range(num_q -2)), num_q -1)  # multi-controlled-toffoli
qc.h(num_q -2)
#o = [0, 1, 2, 4]
qc.mct(list(range(num_q -2)), num_q -2)
#qc.mct(o, num_q -2)
qc.h(num_q -2)
#temp
#qc.h(num_q -2)
#qc.mct(list(range(num_q -2)), num_q -2)  # multi-controlled-toffoli
#qc.h(num_q -2)
#fine temp

qc.h(num_q -1)
for i in range(num_q -2):
    qc.x(i)

oracle_ = qc.to_gate()
oracle_.name = "Oracle"
qc.draw('mpl')

```

```python
#diffuser
def diffuser(nqubits):
    qc = QuantumCircuit(nqubits)
    # Apply transformation |s> -> |00..0> (H-gates)
    for qubit in range(nqubits):
        qc.h(qubit)
    # Apply transformation |00..0> -> |11..1> (X-gates)
    for qubit in range(nqubits):
        qc.x(qubit)
    # Do multi-controlled-Z gate
    qc.h(nqubits-1)
    qc.mct(list(range(nqubits-1)), nqubits-1)  # multi-controlled-toffoli
    qc.h(nqubits-1)
    # Apply transformation |11..1> -> |00..0>
    for qubit in range(nqubits):
        qc.x(qubit)
    # Apply transformation |00..0> -> |s>
    for qubit in range(nqubits):
        qc.h(qubit)
    # We will return the diffuser as a gate
    U_s = qc.to_gate()
    U_s.name = "V"
    return U_s
```

```python
num_qubits = 5
grover_circ = QuantumCircuit(num_qubits)
grover_circ = initialization(grover_circ, range(num_qubits))
for i in range(3):
    grover_circ.append(oracle_, range(num_qubits))
    grover_circ.append(diffuser(num_qubits), range(num_qubits))
grover_circ.measure_all()
grover_circ.draw()
```

```python
grover_circ__ = grover_circ.decompose()
grover_circ__.draw()
```

# Evaluation on classical simulator

```python
aer_sim = Aer.get_backend('aer_simulator')
transpiled_grover_circuit = transpile(grover_circ, aer_sim)
qobj = assemble(transpiled_grover_circuit)
results = aer_sim.run(qobj).result()
counts = results.get_counts()
plot_histogram(counts)
```

```python
from qiskit.tools.monitor import job_monitor
device = backend
transpiled_grover_circuit = transpile(grover_circ, device, optimization_level=3)
job = device.run(transpiled_grover_circuit, shots = 8000)
job_monitor(job, interval=2)
```

    Job Status: job has successfully run

# Results on Quantum Hardware
We can see that the noise in the quantum hardware, considering also the lenght of the circuit given the 5 loops, makes impossible to find the correct solution. This problem could be solved simply considering more desired results, then reducing the loops.

```python
results = job.result()
answer = results.get_counts(grover_circ)
plot_histogram(answer)
```

```python

```

```python

```


.. meta::
   :description: A guide which introduces the most important steps to get started with pymoo, an open-source multi-objective optimization framework in Python.

.. meta::
   :keywords: Multi-objective Optimization, Python, Evolutionary Computation, Optimization Test Problem, Hypervolume

# Preface: Basics and Challenges

Without any loss of generality an optimization problem can be defined by:

\begin{align}
\begin{split}
\min \quad& f_{m}(x) \quad \quad \quad \quad m = 1,..,M  \\[4pt]
\text{s.t.}   \quad& g_{j}(x) \leq 0  \quad \; \; \,  \quad j = 1,..,J \\[2pt]
\quad& h_{k}(x) = 0        \quad  \; \; \quad k = 1,..,K \\[4pt]
\quad& x_{i}^{L} \leq x_{i} \leq x_{i}^{U}  \quad i = 1,..,N \\[2pt]
\quad& x \in \Omega
\end{split}
\end{align}

where $x_i$ represents the $i$-th variable to be optimized, $x_i^{L}$ and $x_i^{U}$ its lower and upper bound, $f_m$ the $m$-th objective function, $g_j$ the $j-th$ inequality constraint and $h_k$ the $k$-th equality constraint.
The objective function(s) $f$ are supposed to be minimized by satisfying all equality and inequality constraints. If a specific objective function is maximized ($\max f_i$), one can redefine the problem to minimize its negative value ($\min -f_i$).

Instead of starting coding your problem immediately, it is recommendable to first think about the mathematical problem formulation. Doing so makes you being aware of the complete optimization problem. This also helps you to identify the challenging facets of your optimization problem and, thus, to select a suitable algorithm. In this guide, we will demonstrate an example of a multi-objective problem, use `pymoo` to obtain a solution set, and theoretically derive the optimum for verification purposes.

If you have some experience solving optimization problems, the following might already be apparent to you. However, if you are new to optimization, thinking about your problem's characteristics is vital. In the following a few common facts and challenges to consider when intending to solve a (real-world) optimization problem are discussed:

**Variable Types.** The variables span the search space $\Omega$ of your optimization problem. Thus, the type of variables is an essential aspect of the problem to be paid attention to. Different variables types, such as continuous, discrete/integer, binary, or permutation, define the characteristics of the search space. In some cases, the variable types might be even mixed, which increases the complexity further.

**Number of Variables.** Not only the type but also the number of variables ($N$) is essential. For either a very small or large number, different algorithms are known to work more efficiently. You can imagine that solving a problem with only ten variables is fundamentally different from solving one with a couple of thousand. For large-scale optimization problems, even the second-order derivate becomes computationally very expensive, and efficiently handling the memory plays a more important role.

**Number of Objectives.** Some optimization problems have more than one conflicting objective ($M>1$) to be optimized. Before researchers have investigated multi-objective optimization, single-objective problems were the main focus. Single-objective optimization is only a particular case where $M=1$. In multi-objective optimization, the solution's domination relation generalizes the comparison of two scalars in single-objective optimization.
Moreover, having more than one dimension in the objective space, the optimum (most of the time) consists of a set of non-dominated solutions.
Because a *set* of solutions should be obtained, population-based algorithms have mainly been used as solvers.

**Constraints.** Optimization problems have two types of constraints, inequality ($g$) and equality ($h$) constraints. From an end-user perspective, constraints have a priority over objective values. No matter how good the solution's objectives are, it is considered infeasible if it turns out to violate just a single constraint. Constraints can have a big impact on the complexity of the problem. For instance, if only a few islands in the search space are feasible or a large number of constraints ($|J|+|K|$) need to be satisfied. For genetic algorithms satisfying equality constraints can be rather challenging. Thus, this needs to be addressed differently, for instance, by mapping the search space to a utility space where the equality constraints are always satisfied or injecting the knowledge of the equality constraint through customization.

**Multi-modality.** Most aspects discussed so far are most likely known or to be relatively easy to define. However, the nature of the fitness landscape is less obvious bet yet essential to be aware of. In the case of multi-modal fitness landscapes, optimization becomes inevitably more difficult due to the existence of a few or even many local optima. For the solution found, one must always ask if the method has explored enough regions in the search space to maximize the probability of obtaining the global optimum. A multi-modal search space quickly shows the limitation of local search, which can easily get stuck.

**Differentiability.** A function being differentiable implies the first or even second-order derivative can be calculated. Differentiable functions allow gradient-based optimization methods to be used, which can be a great advantage over gradient-free methods. The gradient provides a good indication of what direction shall be used for the search. Most gradient-based algorithms are point-by-point based and can be highly efficient for rather unimodal fitness landscapes.
However, in practice, often functions are non-differentiable, or a more complicated function requires a global instead of a local search. The research field addressing problems without knowing their mathematical optimization is also known as black-box optimization.

**Evaluation Time.** Many optimization problems in practice consist of complicated and lengthy mathematical equations or domain-specific software to be evaluated. The usage of third-party software often results in a computationally expensive and time-consuming function for evaluating objectives or constraints. For those types of problems, the algorithm's overhead for determining the next solutions to be evaluated is often neglectable. A commercial software performing an evaluation often comes with various more practical issues such as distributed computing, several instances to be used in parallel and software license, and the software's possible failure for specific design variable combinations.

**Uncertainty.** Often it is assumed that the objective and constraint functions are of a deterministic manner. However, if one or multiple target functions are nondeterministic, this introduces noise or also referred to as uncertainty. One technique to address the underlying randomness is to repeat the evaluation for different random seeds and average the resulting values. Moreover, the standard deviation derived from multiple evaluations can be utilized to determine the performance and the reliability of a specific solution. In general, optimization problems with underlying uncertainty are investigated by the research field called stochastic optimization.

Of course, this shall not be an exhaustive list of problem characteristics but rather an idea of how fundamentally different optimization problems are. Being aware of possible challenges, one can make better decisions regarding a method and its suitability. In this tutorial, solving a constrained bi-objective optimization problem is demonstrated. This, and your problem's characteristics, shall help you to use *pymoo* as a toolbox to tackle your optimization problem.