# Inferring cosmology power spectrum parameters from 2D fields
(by Lucas Makinen and Tom Charnock)

A useful example to consider is the inference of the parameter values controlling the shape of the power spectrum used to generate 2D Gaussian random fields. This is interesting because we can analytically calculate the Fisher information to check our convergence and it has aspects which are very similar to cosmological field analysis. This example is split into two parts, first the analytic comparison when the power spectrum is of the form
$$P(k) = Ak^{-B}$$
and a second part where we use a jax implementation of a cosmology library (`pip install jax-cosmo`) to infer cosmological parameters from 2D Gaussian random fields.

In this example we will also show how inception blocks can be implemented in a stax-like neural model.

## Power law example - $P(k) = Ak^{-B}$

The likelihood for an $N_{\rm pix}\times N_{\rm pix}$ Gaussian field, $\boldsymbol{\delta}$, can be explicitly written down for the Fourier transformed data, $\boldsymbol{\Delta}$ as
$$\mathcal{L}(\boldsymbol{\Delta}|\boldsymbol{\theta}) = \frac{1}{(2\pi)^{N_{\rm pix}^2 / 2} |P({\bf k}, \boldsymbol{\theta})|^{1/2}}\exp{\left(-\frac{1}{2}\boldsymbol{\Delta}\left(P({\bf k}, \boldsymbol{\theta})\right)^{-1}\boldsymbol{\Delta}\right)}$$
Since the Fisher information can be calculated from the expectation value of the second derivative of the score, i.e. the log likelihood
$${\bf F}_{\alpha\beta} = - \left.\left\langle\frac{\partial^2\ln\mathcal{L}(\Delta|\boldsymbol{\theta})}{\partial\theta_\alpha\partial\theta_\beta}\right\rangle\right|_{\boldsymbol{\theta}=\boldsymbol{\theta}^\textrm{fid}}$$
Then we know that analytically the Fisher information must be
$${\bf F}_{\alpha\beta} = \frac{1}{2} {\rm Tr} \left(\frac{\partial P({\bf k}, \boldsymbol{\theta})}{\partial\theta_\alpha}\left(P({\bf k}, \boldsymbol{\theta})\right)^{-1}\frac{\partial P({\bf k}, \boldsymbol{\theta})}{\partial\theta_\beta}\left(P({\bf k}, \boldsymbol{\theta})\right)^{-1}\right)$$
where $\alpha$ and $\beta$ label the parameters $A$ and $B$ in the power spectrum. As each $k$-mode is uncoupled for this power law form then the derivatives  are
$$\begin{align}
\frac{\partial P({\bf k}, \boldsymbol{\theta})}{\partial A} = &~k^{-B}\\
\frac{\partial P({\bf k}, \boldsymbol{\theta})}{\partial B} = & -Ak^{-B}\ln k.\\
\end{align}$$
We can set up these functions. Note that for large fields there can be an issue with numerical overflow and as such we will use jax in 64 bit mode.

```python
%tensorflow_version 2.x
import tensorflow as tf
print(tf.__version__)
```

    2.5.0

```python
!pip install -q jax==0.2.11
```

         |████████████████████████████████| 604kB 3.9MB/s
      Building wheel for jax (setup.py) ... done

```python
!pip install tensorflow-probability
```

    Requirement already satisfied: tensorflow-probability in /usr/local/lib/python3.7/dist-packages (0.12.1)
    Requirement already satisfied: cloudpickle>=1.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (1.3.0)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (1.15.0)
    Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (1.19.5)
    Requirement already satisfied: dm-tree in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (0.1.6)
    Requirement already satisfied: gast>=0.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (0.4.0)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (4.4.2)

```python
import tensorflow_probability as tfp
print(tfp.__version__)
tfd = tfp.distributions
tfb = tfp.bijectors
```

    0.12.1

```python
!pip install -q imnn
```

         |████████████████████████████████| 133kB 2.9MB/s
         |████████████████████████████████| 81kB 4.3MB/s

```python
pip install -q jax-cosmo
```

         |████████████████████████████████| 286kB 4.0MB/s
      Building wheel for jax-cosmo (setup.py) ... done

```python
import jax.numpy as np
from jax import grad, jit, vmap
from jax import random
import jax

#print('jax version: ', jax.__version__)

# for model stuff
import jax.experimental.optimizers as optimizers
import jax.experimental.stax as stax

# tensorflow-prob VANILLA
tfd = tfp.distributions
tfb = tfp.bijectors

# tensorflow-prob-JAX
import tensorflow_probability.substrates.jax as tfpj

tfdj = tfpj.distributions
tfbj = tfpj.bijectors

# matplotlib stuff
import matplotlib.pyplot as plt
from scipy.linalg import toeplitz
import seaborn as sns
sns.set()

# for imnn
import imnn
import imnn.lfi

rng = random.PRNGKey(2)
```

```python
from jax.config import config
config.update('jax_enable_x64', True)
```

```python
from jax.lib import xla_bridge
print(xla_bridge.get_backend().platform)
```

    gpu

First we'll set up our $P(k)$

```python
def P(k, A=1, B=0):
    return A * k ** (-B)
```

And the log likelihood is then

```python
def log_likelihood(k, A, B, Δ, N):
    Δ = Δ[1:N // 2, 1:N // 2].flatten()
    k = k[1:N // 2, 1:N // 2].flatten()
    dlength = len(k)
    def fn(_A, _B):
        nrm = np.pad(np.ones(dlength - 2) * 2, (1, 1), constant_values=1.)
        nrm = jax.ops.index_update(
          nrm, np.array([[0],[(dlength - 2)]]), np.array([[1], [1]]))
        nrm = 1
        powers = P(k, A=_A, B=_B)
        C = powers * nrm
        invC = 1. / powers
        exponent = - 0.5 * np.sum(np.conj(Δ) * invC * Δ)
        norm = -0.5 * np.sum(np.log(C)) -0.5 * len(Δ) * np.log(np.pi * 2.)
        return (exponent + norm)
    return jax.vmap(fn)(A, B)
```

and the Fisher information matrix is given by

```python
def fisher(θ, k, N):
    A, B = θ
    k = k[1:N // 2, 1:N // 2].flatten()
    Pk = P(k, A, B)

    Cinv = np.diag(1. / Pk)
    C_A =  np.diag(k ** -B)
    C_B =  np.diag(- Pk * np.log(k))

    F_AA = 0.5 * np.trace((C_A @ Cinv @ C_A @ Cinv))
    F_AB = 0.5 * np.trace((C_A @ Cinv @ C_B @ Cinv))
    F_BA = 0.5 * np.trace((C_B @ Cinv @ C_A @ Cinv))
    F_BB = 0.5 * np.trace((C_B @ Cinv @ C_B @ Cinv))

    return np.array([[F_AA, F_AB], [F_BA, F_BB]])
```

Notice that we only take the unique modes to calculate the Fisher information. To actually set up our 2D $k$-modes we stack values from $0\to N_i/2 + 1$ to values from $-N_i/2+1\to0$ and normalising them by $2\pi/N_i$ where $N_i$ is the number of $k$-modes in each axis. If we choose that our fields have $128\times128$ pixels then we can calculate $k$ as

```python
N = 128
shape = (N, N)

k = np.sqrt(
    np.sum(
        np.array(
            np.meshgrid(
                *((np.hstack(
                    (np.arange(0, _shape // 2 + 1),
                     np.arange(-_shape // 2 + 1, 0)))
                   * 2 * np.pi / _shape)**2.
                  for _shape in shape))),
        axis=0))
```

For a fiducial $A^\textrm{fid}=1$ and $B^\textrm{fid}=1/2$ we get a Fisher information matrix equal to

```python
θ_fid = np.array([1., 0.5], dtype=np.float32)
n_params = θ_fid.shape[0]

F = fisher(θ_fid, k, N)
print(f"F = {F}")
print(f"det(F) = {np.linalg.det(F)}")
```

    F = [[ 1984.5        -1545.06379524]
     [-1545.06379524  1690.49264978]]
    det(F) = 967560.5321330648

### 2D Gaussian random field simulator in jax

To create a 2D Gaussian random field we can follow these steps:

1. Generate a $(N_\textrm{pix}\times N_\textrm{pix})$ white noise field $\varphi$ such that $\langle \varphi_k \varphi_{-k} \rangle' = 1$

2. Fourier Transform $\varphi$ to real space: $R_{\rm white}({\bf x}) \rightarrow R_{\rm white}({\bf k})$
   Note that NumPy's DFT Fourier convention is:
   $$\phi_{ab}^{\bf k} = \sum_{c,d = 0}^{N-1} \exp{(-i x_c k_a - i x_d k_b) \phi^{\bf x}_{cd}}$$
   $$\phi_{ab}^{\bf x} = \frac{1}{N^2}\sum_{c,d = 0}^{N-1} \exp{(-i x_c k_a - i x_d k_b) \phi^{\bf k}_{cd}}$$

3. Scale white noise $R_{\rm white}({\bf k})$ by the chosen power spectrum evaluated over a field of $k$ values:
   $$R_P({\bf k}) = P^{1/2}(k) R_{\rm white}({\bf k}) $$
   Here we need to ensure that this array of amplitudes are Hermitian, e.g. $\phi^{* {\bf k}}_{a(N/2 + b)} = \phi^{{\bf k}}_{a(N/2 - b)}$. This is accomplished by choosing indices $k_a = k_b = \frac{2\pi}{N} (0, \dots, N/2, -N/2+1, \dots, -1)$ (as above) and then evaluating the square root of the outer product of the meshgrid between the two: $k = \sqrt{k^2_a + k^2_b}$. We can then evaluate $P^{1/2}(k)$.

4. Fourier Transform $R_{P}({\bf k})$ to real space: $R_P({\bf x}) = \int d^d \tilde{k} e^{i{\bf k} \cdot {\bf x}} R_p({\bf k})$
   $$R_{ab}^{\bf x} = \frac{1}{N^2}\sum_{c,d = 0}^{N-1} \exp{(-i x_c k_a - i x_d k_b) R^{\bf k}_{cd}}$$

We are going to build a broadcastable jax simultor which takes in a variety of different shaped parameter arrays and vmaps them until a single parameter pair are passed. This is very efficient for performing the ABC for example.

```python
def simulator(rng, θ, simulator_args, foregrounds=None):
    def fn(rng, A, B):
        dim = len(simulator_args["shape"])
        L = simulator_args["L"]
        if np.isscalar(L):
            L = [L] * int(dim)
        Lk = ()
        shape = ()
        for i, _shape in enumerate(simulator_args["shape"]):
            Lk += (_shape / L[i],)
            if _shape % 2 == 0:
                shape += (_shape + 1,)
            else:
                shape += (_shape,)

        k = simulator_args["k"]
        k_shape = k.shape
        k = k.flatten()[1:]
        tpl = ()
        for _d in range(dim):
            tpl += (_d,)

        V = np.prod(np.array(L))
        scale = V**(1. / dim)
        fft_norm = np.prod(np.array(Lk))

        rng, key = jax.random.split(rng)

        mag = jax.random.normal(
            key, shape=shape)
        pha = 2. * np.pi * jax.random.uniform(
            key, shape=shape)

        # now make hermitian field (reality condition)
        revidx = (slice(None, None, -1),) * dim
        mag = (mag + mag[revidx]) / np.sqrt(2)
        pha = (pha - pha[revidx]) / 2 + np.pi
        dk = mag * (np.cos(pha) + 1j * np.sin(pha))
        cutidx = (slice(None, -1),) * dim
        dk = dk[cutidx]

        powers = np.concatenate(
            (np.zeros(1),
             np.sqrt(P(k, A=A, B=B)))).reshape(k_shape)

        if simulator_args['vol_norm']:
            powers /= V

        if simulator_args["log_normal"]:
            powers = np.real(
                np.fft.ifftshift(
                    np.fft.ifftn(
                        powers)
                    * fft_norm) * V)

            powers = np.log(1. + powers)
            powers = np.abs(np.fft.fftn(powers))

        fourier_field = powers * dk
        fourier_field = jax.ops.index_update(
            fourier_field,
            np.zeros(dim, dtype=int),
            np.zeros((1,)))

        if simulator_args["log_normal"]:
            field = np.real(np.fft.ifftn(fourier_field)) * fft_norm * np.sqrt(V)
            sg = np.var(field)
            field = np.exp(field - sg / 2.) - 1.

        else:
            field = np.real(np.fft.ifftn(fourier_field) * fft_norm * np.sqrt(V)**2)

        if simulator_args["N_scale"]:
            field *= scale

        if foregrounds is not None:
            rng, key = jax.random.split(key)
            foreground = foregrounds[
                jax.random.randint(
                    key,
                    minval=0,
                    maxval=foregrounds.shape[0],
                    shape=())]
            field = np.expand_dims(field + foreground, (0,))

        if not simulator_args["squeeze"]:
            field = np.expand_dims(field, (0, 1))

        return np.array(field, dtype='float32')

    if isinstance(θ, tuple):
        A, B = θ
    else:
        A = np.take(θ, 0, axis=-1)
        B = np.take(θ, 1, axis=-1)
    if A.shape == B.shape:
        if len(A.shape) == 0:
            return fn(rng, A, B)
        else:
            keys = jax.random.split(rng, num=A.shape[0] + 1)
            rng = keys[0]
            keys = keys[1:]
            return jax.vmap(
                lambda key, A, B: simulator(
                    key, (A, B), simulator_args=simulator_args))(
                keys, A, B)
    else:
        if len(A.shape) > 0:
            keys = jax.random.split(rng, num=A.shape[0] + 1)
            rng = keys[0]
            keys = keys[1:]
            return jax.vmap(
                lambda key, A: simulator(
                    key, (A, B), simulator_args=simulator_args))(
                keys, A)
        elif len(B.shape) > 0:
            keys = jax.random.split(rng, num=B.shape[0])
            return jax.vmap(
                lambda key, B: simulator(
                    key, (A, B), simulator_args=simulator_args))(
                keys, B)
```

We can now set the simulator arguments, i.e. the $k$-modes to evaluate, the length of the side of a box, the shape of the box and whether to normalise via the volume and squeeze the output dimensions

```python
simulator_args = dict(
    k=k,
    L=N,
    shape=shape,
    vol_norm=True,
    N_scale=True,
    squeeze=True,
    log_normal=False)
```

Now we can simulate some target data at, for example, $A^\textrm{target}=0.7$ and $B^\textrm{target}=0.8$:

```python
θ_target = np.array([0.7, 0.8])

rng, key = jax.random.split(rng)
δ_target = simulator(key, θ_target, simulator_args=simulator_args)
plt.imshow(δ_target)
plt.colorbar();
```

We can now define our prior distribution (in this case a uniform distribution over $A$ and $B$) with values between 0.1 and 1.25 for both parameters

```python
prior = tfpj.distributions.Blockwise(
    [tfpj.distributions.Uniform(low=low, high=high)
     for low, high in zip([0.1, 0.1], [1.25, 1.25])])
prior.low = np.array([0.1, 0.1])
prior.high = np.array([1.25, 1.25])
```

To evaluate the likelihood of this field we can now use (dividing the target $\delta$ by $N$ to remove added scaling)

```python
LFI = imnn.lfi.LikelihoodFreeInference(
    prior=prior,
    gridsize=100)
A, B = np.meshgrid(*LFI.ranges)
LFI.n_targets=1
LFI.put_marginals(
        jax.nn.softmax(
            np.real(
                log_likelihood(
                    k,
                    A.ravel(),
                    B.ravel(),
                    np.fft.fftn(δ_target / N),
                    N)
                ),axis=0
            ).reshape((100, 100)).T[np.newaxis]);
LFI.marginal_plot(
    known=θ_target,
    label="Analytic likelihood",
    axis_labels=["A", "B"]);
```

```python
#@title sanity check [don't have to run]
fig,ax = plt.subplots(2,2)

for i in range(20):
    θ_target = np.array([0.7, 0.8])

    rng, key = jax.random.split(rng)
    δ_target = simulator(key, θ_target, simulator_args=simulator_args)

    LFI = imnn.lfi.LikelihoodFreeInference(
        prior=prior,
        gridsize=100)
    A, B = np.meshgrid(*LFI.ranges)
    LFI.n_targets=1
    LFI.put_marginals(
        jax.nn.softmax(
            np.real(
                log_likelihood(
                    k,
                    A.ravel(),
                    B.ravel(),
                    np.fft.fftn(δ_target / N),
                    N)
                ), axis=0).reshape((100, 100)).T[np.newaxis]);
    ax = LFI.marginal_plot(
        ax=ax,
        known=θ_target,
        label="Analytic likelihood",
        axis_labels=["A", "B"]);

plt.show()
```

## Training an IMNN

Now lets train an IMNN to summaries such Gaussian random fields to see how much information we can extract an what sort of constraints we can get. We will use 5000 simulations to estimate the covariance and use all of their derivatives and we'll summarise the whole random Gaussian field by 2 summaries.

```python
n_s = 2000
n_d = n_s

n_summaries = 2
```

We're going to use a fully convolutional inception network built using stax with some custom designed blocks. The inception block itself is implemented as

```python
def InceptBlock(filters, strides, do_5x5=True, do_3x3=True):
    """InceptNet convolutional striding block.
    filters: tuple: (f1,f2,f3)
    filters1: for conv1x1
    filters2: for conv1x1,conv3x3
    filters3L for conv1x1,conv5x5"""

    filters1, filters2, filters3 = filters
    conv1x1 = stax.serial(stax.Conv(filters1, (1, 1), strides, padding="SAME"))

    filters4 = filters2
    conv3x3 = stax.serial(stax.Conv(filters2, (1, 1), strides=None, padding="SAME"),
                        stax.Conv(filters4, (3, 3), strides, padding="SAME"))

    filters5 = filters3
    conv5x5 = stax.serial(stax.Conv(filters3, (1, 1), strides=None, padding="SAME"),
                         stax.Conv(filters5, (5, 5), strides, padding="SAME"))

    maxpool = stax.serial(stax.MaxPool((3, 3), padding="SAME"),
                         stax.Conv(filters4, (1, 1), strides, padding="SAME"))

    if do_3x3:
        if do_5x5:
            return stax.serial(
                  stax.FanOut(4),
                  stax.parallel(conv1x1, conv3x3, conv5x5, maxpool),
                  stax.FanInConcat(),
                  stax.LeakyRelu)
        else:
            return stax.serial(
                  stax.FanOut(3),
                  stax.parallel(conv1x1, conv3x3, maxpool),
                  stax.FanInConcat(),
                  stax.LeakyRelu)
    else:
        return stax.serial(
              stax.FanOut(2),
              stax.parallel(conv1x1, maxpool),
              stax.FanInConcat(),
              stax.LeakyRelu)
```

We'll also want to make sure that the output of the network is the correct shape, for which we'll introduce a Reshaping layer

```python
def Reshape(shape):
    """Layer function for a reshape layer."""
    init_fun = lambda rng, input_shape: (shape,())
    apply_fun = lambda params, inputs, **kwargs: np.reshape(inputs, shape)
    return init_fun, apply_fun
```

Now we can build the network, with kernel sizes of 4 in each direction in each layer

```python
fs = 64

model = stax.serial(
        InceptBlock((fs, fs, fs), strides=(4, 4)),
        InceptBlock((fs, fs, fs), strides=(4, 4)),
        InceptBlock((fs, fs, fs), strides=(4, 4)),
        InceptBlock((fs, fs, fs), strides=(2, 2), do_5x5=False, do_3x3=False),
        stax.Conv(n_summaries, (1, 1), strides=(1, 1), padding="SAME"),
        stax.Flatten,
        Reshape((n_summaries,)))
```

We'll also grab an adam optimiser from jax.experimental.optimizers

```python
optimiser = optimizers.adam(step_size=1e-3)
```

Note that due to the form of the network we'll want to have simulations that have a "channel" dimension, which we can set up by not allowing for squeezing in the simulator.

### Initialise IMNN

Finally we can initialise the IMNN, letting the IMNN module decide what type of IMNN subclass will be used (we'll be using SimulatorIMNN)

```python
rng, key = jax.random.split(rng)
IMNN = imnn.IMNN(
        n_s=n_s,
        n_d=n_d,
        n_params=n_params,
        n_summaries=n_summaries,
        input_shape=(1, 1) + shape,
        θ_fid=θ_fid,
        model=model,
        optimiser=optimiser,
        key_or_state=key,
        simulator=lambda rng, θ: simulator(
            rng, θ, simulator_args={
                **simulator_args,
                **{"squeeze": False}}))
```

    `simulator` provided, using SimulatorIMNN

And finally we can fit the IMNN (we'll use generic regularisation parameters of $\lambda=10$ and $\epsilon=0.1$) and allow early stopping to determine the end of fitting.

```python
%%time
rng, key = jax.random.split(rng)
IMNN.fit(λ=10., ϵ=0.1, rng=key, print_rate=None, min_iterations=500, best=True)
```

    CPU times: user 13min 48s, sys: 6.06 s, total: 13min 54s
    Wall time: 14min 2s

```python
IMNN.plot(expected_detF=np.linalg.det(F));
```

```python
np.linalg.det(IMNN.F) / np.linalg.det(F)
```

    DeviceArray(0.97123594, dtype=float64)

```python
np.linalg.det(F)
```

    DeviceArray(967560.53213307, dtype=float64)

## Inference

We can now attempt to do inference of some target data using the IMNN. The first thing we should do is make a Gaussian approximation using a parameter estimate from the IMNN and the Fisher information reached at the end of fitting. Note that since the fiducial parameter values are far from the "target" that this estimate of the Fisher information as the covariance will likely be misleading.

```python
GA = imnn.lfi.GaussianApproximation(
    parameter_estimates=IMNN.get_estimate(np.expand_dims(δ_target, (0, 1, 2))),
    invF=np.expand_dims(np.linalg.inv(IMNN.F), 0),
    prior=prior,
    gridsize=100)
GA.marginal_plot(
    known=θ_target,
    label="Gaussian approximation",
    axis_labels=["A", "B"],
    colours="C1");
```

And finally we can do an approximate Bayesian computation

```python
ABC = imnn.lfi.ApproximateBayesianComputation(
    target_data=np.expand_dims(δ_target, (0, 1, 2)),
    prior=prior,
    simulator=lambda rng, θ: simulator(rng, θ, simulator_args={**simulator_args, **{'squeeze':False}}),
    compressor=IMNN.get_estimate,
    gridsize=100,
    F=np.expand_dims(IMNN.F, 0))
```

```python
rng, key = jax.random.split(rng)
ABC(ϵ=1., rng=key, n_samples=10000, min_accepted=1000,
    smoothing=1, max_iterations=1000);
```

    [1010] accepted in last  43 iterations  (430000 simulations done).

```python
ax = LFI.marginal_plot(
    known=θ_target,
    label="Analytic likelihood",
    axis_labels=["A", "B"])

GA.marginal_plot(
    ax=ax,
    label="Gaussian approximation",
    colours="C1",
    axis_labels=["A", "B"])

ABC.marginal_plot(
    ax=ax,
    label="Approximate Bayesian computation",
    colours="C2");

plt.show()

```


<p align="center">

</p>

## Data Analytics

### Fitting Parametric Distributions  in Python

#### Michael Pyrcz, Associate Professor, The University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Data Analytics: Fitting Parametric Distributions

Here's a demonstration of fitting parametric distributions in Python. This demonstration is part of the resources that I include for my courses in Spatial / Subsurface Data Analytics at the Cockrell School of Engineering at the University of Texas at Austin.

#### Parametric Distributions

We will cover fitting the following distribution:

* Gaussian

For more information about working with parametric distributions in Python see this [demonstration](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/PythonDataBasics_ParametricDistributions.ipynb).

I have a lecture on these parametric distributions available on [YouTube](https://www.youtube.com/watch?v=U7fGsqCLPHU&t=1687s).

#### Fitting Parametric Distributions

With distribution fitting we are maximizing the likelihood function, $L(\theta)$:

\begin{equation}
L(\theta) = \prod^{n}_{\alpha=1} f\left(x_{\alpha} | \theta \right)
\end{equation}

the product sum of the probablity of the data, $f\left(x_{\alpha}\right)$ over all the data $\alpha = 1,\ldots,n$, given the set of distribution parameters, $\theta$.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - unconv_MV_v4.csv at https://git.io/fhHLT.

#### Importing Packages

We will need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
import math                               # trigonometry etc.
import random                             # for randon numbers
```

#### Set the Working Directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

### Guassian Distribution

Let's now use the Gaussian parametric distribution.

* we will need the parameters mean and the variance

We will apply the forward and reverse operations and calculate the summary statistics.

```python
from scipy.stats import norm as my_dist                                      # import traingular dist as my_dist
dist_type = 'Gaussian'                                                       # give the name of the distribution for labels
mean = 0.15; stdev = 0.03                                                    # given the distribution parameters

x_values = np.linspace(0.0,0.3,100)                                          # get an array of x values
p_values = my_dist.pdf(x_values, loc = mean, scale = stdev)                  # calculate density for each x value
P_values = my_dist.cdf(x_values, loc = mean, scale = stdev)                  # calculate cumulative probablity for each x value

plt.subplot(1,3,1)                                                           # plot the resulting PDF
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3); plt.title('Sampling ' + str(dist_type) + ' PDF'); plt.xlabel('Values'); plt.ylabel('Density')

plt.subplot(1,3,2)                                                           # plot the resulting CDF
plt.plot(x_values, P_values,'r-', lw=5, alpha=0.3); plt.title('Sampling ' + str(dist_type) + ' CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

p_values = np.linspace(0.00001,0.99999,100)                                  # get an array of p-values
x_values = my_dist.ppf(p_values, loc = mean, scale = stdev)                  # apply inverse to get x values from p-values
plt.subplot(1,3,3)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3, label='uniform pdf')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.8, top=0.8, wspace=0.2, hspace=0.3); plt.title('Sampling Inverse ' + str(dist_type) + ' CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

print('The mean is ' + str(round(my_dist.mean(loc = mean, scale = stdev),4)) + '.') # calculate stats and symmetric interval
print('The variance is ' + str(round(my_dist.var(loc = mean, scale = stdev),4)) + '.')

```

#### Fitting the Distribution

Let's make a random dataset and then fit it with a Gaussian distribution.

```python
n = 100
samples = my_dist.rvs(size=int(n), loc = mean, scale = stdev, random_state = 73073).tolist() # make a synthetic dataset

plt.subplot(1,2,1)                                                 # plot the binned PDF
plt.hist(samples,bins = np.linspace(mean-3*stdev,mean+3*stdev,30), density = True, color = 'red', edgecolor = 'black', alpha = 0.2)
plt.xlim(0.0,0.3); plt.ylim(0,25); plt.xlabel('Values'); plt.ylabel('Density'); plt.title('Synthetic Sample Data')

plt.subplot(1,2,2)                                                 # plot the CDF
plt.hist(samples,bins = np.linspace(mean-3*stdev,mean+3*stdev,30), density = True, color = 'red', edgecolor = 'black', cumulative = True, histtype = 'stepfilled',alpha = 0.2)
plt.xlim(0.0,0.3); plt.ylim(0,1); plt.xlabel('Values'); plt.ylabel('Cumulative Probability'); plt.title('Synthetic Sample Data')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.0, wspace=0.2, hspace=0.3)
```

```python
plt.subplot(1,2,1)                                                 # plot the binned PDF
plt.hist(samples,bins = np.linspace(mean-3*stdev,mean+3*stdev,30), density = True, color = 'red', edgecolor = 'black', alpha = 0.2)

x = np.linspace(my_dist.ppf(0.0001, loc = mean, scale = stdev),my_dist.ppf(0.9999, loc = mean, scale = stdev), 100)

fit_mean, fit_stdev = my_dist.fit(samples,loc = mean, scale = stdev) # fit MLE of the distribution parameters
print('MLE mean = ' + str(round(fit_mean,3)) + ', MLE standard deviation = ' + str(round(fit_stdev,3)))

plt.plot(x, my_dist.pdf(x, loc = fit_mean, scale = fit_stdev), 'r-', lw=5, alpha=0.6, label='norm pdf')
plt.xlim(0.0,0.3); plt.ylim(0,25); plt.xlabel('Values'); plt.ylabel('Density'); plt.title('Synthetic Sample Data and Parametric Distribution Fit')

plt.subplot(1,2,2)                                                 # plot the binned PDF with the fit parametric distribution
plt.hist(samples,bins = np.linspace(fit_mean-3*fit_stdev,fit_mean+3*fit_stdev,30), density = True, color = 'red', edgecolor = 'black', cumulative = True, histtype = 'stepfilled',alpha = 0.2)
plt.xlim(0.0,0.3); plt.ylim(0,1); plt.xlabel('Values'); plt.ylabel('Cumulative Probability'); plt.title('Synthetic Sample Data and Parametric Distribution Fit')
                                                                   # plot the CDF with the fit parametric distribution
plt.plot(x, my_dist.cdf(x, loc = fit_mean, scale = fit_stdev), 'r-', lw=5, alpha=0.6, label='norm pdf')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.0, wspace=0.2, hspace=0.3)
```

There are many other parametric distributions that we could have included, I just wanted to provide a concise demonstration of distribution fitting.

#### Comments

This was a basic demonstration of fitting parametric distributions.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at [Python Demos](https://github.com/GeostatsGuy/PythonNumericalDemos) and a Python package for data analytics and geostatistics at [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# Plotting radial hydrogen orbitals with sympy
This notebook, loosely inspired on https://ojensen.wordpress.com/2010/08/10/fast-ufunc-ish-hydrogen-solutions/, plots some radial hydrogen functions using sympy, as support for Chapter 2 of the Building Blocks of Matter course at Leiden University.

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
%matplotlib inline

from sympy.physics.hydrogen import R_nl
from sympy import var, simplify
from sympy import init_printing
from sympy.utilities.lambdify import lambdify
from sympy import integrate, oo
from scipy.constants import hbar, alpha, Rydberg, c
```

```python
init_printing()

plt.rcParams['axes.xmargin'] = 0
plt.rcParams['figure.figsize'] = [8,6]

cmap = matplotlib.cm.get_cmap('tab20c')
```

```python
var('n l r')
R_nl(2,1,r,1)*r**2
```

```python
x_np = np.linspace(0, 20, 200)
linestyles = ['-', '--', '-.', ':']

Z = 1

for n_np in range(1,4):
    for l_np in range(0,n_np):
        plt.plot(x_np,lambdify(r, R_nl(n_np, l_np, r, Z))(x_np),
                 color=cmap(4*n_np-4+l_np),
                 linestyle=linestyles[l_np],
                 label="$n = {}, l = {}$".format(n_np, l_np))
plt.ylim([-0.12, 0.18])
plt.xlabel('radial distance ($a_0$)')
plt.ylabel('$R_{nl}$')
plt.legend()
plt.show()
```

```python
x_np = np.linspace(0, 30, 300)
for n_np in range(1,5):
    for l_np in range(0,n_np):
        expected_r = integrate(r * R_nl(n_np, l_np, r, 1)**2 * r**2, (r,0,oo))
        plt.plot(x_np, lambdify(r, R_nl(n_np, l_np, r, 1)**2 * r**2)(x_np),
                 color=cmap(4*(n_np-1) + l_np),
                 linestyle=linestyles[l_np],
                 label=r"$n = {}, l = {}, \langle r \rangle = {}$".format(n_np,
                                                                           l_np,
                                                                           expected_r))
plt.xlabel('radial distance ($a_0$)')
plt.ylabel('Probability')
plt.legend()
plt.show()
```

```python
# Investigate the Transition amplitudes for the sharp and diffuse transitions
diffuse = R_nl(n, 1, r, 1) * r * R_nl(n+1, 2, r, 1) * r**2
sharp = R_nl(n, 1, r, 1) * r * R_nl(n+1, 0, r, 1) * r**2
display(diffuse)
display(sharp)
```

```python
for n_np in [2, 3, 4]:
    plt.plot(x_np*2, lambdify(r, diffuse.subs({n: n_np}))(x_np*2),
             color=cmap(4*(n_np-1)),
             label='Diffuse {} -> {}'.format(n_np, n_np+1))
    plt.plot(x_np*2, lambdify(r, sharp.subs({n: n_np}))(x_np*2),
             color=cmap(4*(n_np-1)),
             linestyle='--',
             label='Sharp {} -> {}'.format(n_np, n_np+1))
plt.xlabel('radial distance ($a_0$)')
plt.legend()
plt.show()
```

```python
for n_np in [2, 3, 4]:
    print("Transition amplitude diffuse {} -> {}: {:2f}".format(n_np,
                                                                n_np+1,
                                                                float(integrate(diffuse.subs({n: n_np}), (r,0,oo)))))
    print("Transition amplitude sharp {} -> {}: {:2f}".format(n_np,
                                                              n_np+1,
                                                              float(integrate(sharp.subs({n: n_np}), (r,0,oo)))))
```

    Transition amplitude diffuse 2 -> 3: 4.747992
    Transition amplitude sharp 2 -> 3: 0.938404
    Transition amplitude diffuse 3 -> 4: 7.565411
    Transition amplitude sharp 3 -> 4: 2.443534
    Transition amplitude diffuse 4 -> 5: 11.038943
    Transition amplitude sharp 4 -> 5: 4.600278


# Exercises

## Exercise 1: Simulate waves on a non-homogeneous string

Simulate waves on a string that consists of two materials with
different density. The tension in the string is constant, but the
density has a jump at the middle of the string.  Experiment with
different sizes of the jump and produce animations that visualize the
effect of the jump on the wave motion.

**Hint.**
According to the section [wave:app:string](#wave:app:string),
the density enters the mathematical model as $\varrho$ in
$\varrho u_{tt} = Tu_{xx}$, where $T$ is the string tension. Modify, e.g., the
`wave1D_u0v.py` code to incorporate the tension and two density values.
Make a mesh function `rho` with density values at each spatial mesh point.
A value for the tension may be 150 N. Corresponding density values can
be computed from the wave velocity estimations in the `guitar` function
in the `wave1D_u0v.py` file.

Filename: `wave1D_u0_sv_discont`.

## Exercise 2: Simulate damped waves on a string

Formulate a mathematical model for damped waves on a string.
Use data from the section [wave:pde1:guitar:data](#wave:pde1:guitar:data), and
tune the damping parameter so that the string is very close to
the rest state after 15 s. Make a movie of the wave motion.
Filename: `wave1D_u0_sv_damping`.

## Exercise 3: Simulate elastic waves in a rod

A hammer hits the end of an elastic rod. The exercise is to simulate
the resulting wave motion using the model ([wave:app:elastic:rod:eq](#wave:app:elastic:rod:eq))
from the section [wave:app:elastic:rod](#wave:app:elastic:rod). Let the rod have length
$L$ and let the boundary $x=L$ be stress free so that $\sigma_{xx}=0$,
implying that $\partial u/\partial x=0$. The left end $x=0$ is
subject to a strong stress pulse (the hammer), modeled as

$$
\sigma_{xx}(t) = \left\lbrace\begin{array}{ll}
S,& 0 < t \leq t_s,\\
0, & t > t_s
\end{array}\right.
$$

The corresponding condition on $u$ becomes $u_x= S/E$
for $t\leq t_s$ and zero afterwards (recall that
$\sigma_{xx} = Eu_x$). This is a non-homogeneous
Neumann condition, and you will need to approximate this condition
and combine it with the scheme (the ideas and manipulations follow
closely the handling of a non-zero initial condition
$u_t=V$ in wave PDEs or the corresponding
second-order ODEs for vibrations).
Filename: `wave_rod`.

## Exercise 4: Simulate spherical waves

Implement a model for spherically symmetric waves using the method
described in the section [wave:app:spherical](#wave:app:spherical). The boundary condition
at $r=0$ must be $\partial u/\partial r=0$, while the condition at
$r=R$ can either be $u=0$ or a radiation condition as described
in [wave:app:exer:radiationBC](#wave:app:exer:radiationBC). The $u=0$ condition is
sufficient if $R$ is so large that the amplitude of the spherical
wave has become insignificant. Make movie(s) of the case where the source
term is located around $r=0$ and sends out pulses

$$
f(r,t) = \left\lbrace\begin{array}{ll}
Q\exp{(-\frac{r^2}{2\Delta r^2})}\sin\omega t,& \sin\omega t\geq 0\\
0, & \sin\omega t < 0
\end{array}\right.
$$

Here, $Q$ and $\omega$ are constants to be chosen.

**Hint.**
Use the program `wave1D_u0v.py` as a starting point. Let `solver`
compute the $v$ function and then set $u=v/r$. However,
$u=v/r$ for $r=0$ requires special treatment. One possibility is
to compute `u[1:] = v[1:]/r[1:]` and then set `u[0]=u[1]`. The latter
makes it evident that $\partial u/\partial r = 0$ in a plot.

Filename: `wave1D_spherical`.

## Problem 5: Earthquake-generated tsunami over a subsea hill

A subsea earthquake leads to an immediate lift of the water surface,
see [Figure](#wave:app:fig:1D:tsunami:flat).  The lifted water
surface splits into two tsunamis, one traveling to the right and one
to the left, as depicted in [Figure](#wave:app:fig:1D:tsunami:2waves).  Since tsunamis are normally very
long waves, compared to the depth, with a small amplitude, compared to
the wave length, a standard wave equation is relevant:

$$
\eta_{tt} = (gH(x)\eta_x)_x,
$$

where $\eta$ is the elevation of the water surface,
$g$ is the acceleration of gravity, and $H(x)$ is the still water depth.

Sketch of initial water surface due to a subsea earthquake.

An initial surface elevation is split into two waves.

To simulate the right-going tsunami, we can impose a symmetry boundary
at $x=0$: $\partial\eta/\partial x =0$.  We then simulate the wave
motion in $[0,L]$. Unless the ocean ends at $x=L$, the waves should
travel undisturbed through the boundary $x=L$. A radiation condition
as explained in [wave:app:exer:radiationBC](#wave:app:exer:radiationBC) can be used for
this purpose. Alternatively, one can just stop the simulations before
the wave hits the boundary at $x=L$. In that case it does not matter
what kind of boundary condition we use at $x=L$. Imposing $\eta =0$
and stopping the simulations when $|\eta_i^n| > \epsilon$, $i=N_x-1$,
is a possibility ($\epsilon$ is a small parameter).

The shape of the initial surface can be taken as a
Gaussian function,

$$
\begin{equation}
I(x;I_0,I_a,I_m,I_s) =
I_0 + I_a\exp{\left(-\left(\frac{x-I_m}{I_s}\right)^2\right)},
\label{_auto1} \tag{1}
\end{equation}
$$

with $I_m=0$ reflecting the location of the peak of $I(x)$ and
$I_s$ being a measure of the width of the function $I(x)$
($I_s$ is $\sqrt{2}$ times the standard deviation of the familiar
normal distribution curve).

Now we extend the problem with a hill at the sea bottom, see [Figure](#wave:app:fig:1D:tsunami:hill). The wave speed $c=\sqrt{gH(x)} =
\sqrt{g(H_0-B(x))}$ will then be reduced in the shallow water above
the hill.

Sketch of an earthquake-generated tsunami passing over a subsea hill.

One possible form of the
hill is a Gaussian function,

$$
\begin{equation}
B(x;B_0,B_a,B_m,B_s) =
B_0 + B_a\exp{\left(-\left(\frac{x-B_m}{B_s}\right)^2\right)},
\label{wave:app:exer:tsunami1D:hill:Gauss} \tag{2}
\end{equation}
$$

but many other shapes are also possible, e.g., a "cosine hat" where

$$
\begin{equation}
B(x; B_0, B_a, B_m, B_s) = B_0 + B_a\cos{\left( \pi\frac{x-B_m}{2B_s}\right)},
\label{wave:app:exer:tsunami1D:hill:cohat} \tag{3}
\end{equation}
$$

when $x\in [B_m - B_s, B_m + B_s]$ while $B=B_0$ outside this
interval.

Also an abrupt construction may be tried:

$$
\begin{equation}
B(x; B_0, B_a, B_m, B_s) = B_0 + B_a,
\label{wave:app:exer:tsunami1D:hill:box} \tag{4}
\end{equation}
$$

for $x\in [B_m - B_s, B_m + B_s]$ while $B=B_0$ outside this
interval.

The [`wave1D_dn_vc.py`](${src_wave}/wave1D/wave1D_dn_vc.py)
program can be used as starting point for the implementation.
Visualize both the bottom topography and the
water surface elevation in
the same plot.
Allow for a flexible choice of bottom shape:
([2](#wave:app:exer:tsunami1D:hill:Gauss)),
([3](#wave:app:exer:tsunami1D:hill:cohat)),
([4](#wave:app:exer:tsunami1D:hill:box)), or $B(x)=B_0$ (flat).

The purpose of this problem is to explore the quality of the numerical
solution $\eta^n_i$ for different shapes of the bottom obstruction.
The "cosine hat" and the box-shaped hills have abrupt changes in the
derivative of $H(x)$ and are more likely to generate numerical noise
than the smooth Gaussian shape of the hill. Investigate if this is true.
Filename: `tsunami1D_hill`.

## Problem 6: Earthquake-generated tsunami over a 3D hill

This problem extends [Problem 5: Earthquake-generated tsunami over a subsea hill](#wave:app:exer:tsunami1D:hill)
to a three-dimensional wave phenomenon, governed by the 2D PDE

$$
\begin{equation}
\eta_{tt} = (gH\eta_x)_x + (gH\eta_y)_y = \nabla\cdot (gH\nabla\eta)
\label{wave:app:sw:2D:eta:2ndoeq2} \tag{5}
\thinspace .
\end{equation}
$$

We assume that the earthquake arises from a fault along the line $x=0$
in the $xy$-plane so that the initial lift of the surface can be taken
as $I(x)$ in [Problem 5: Earthquake-generated tsunami over a subsea hill](#wave:app:exer:tsunami1D:hill). That is, a
plane wave is propagating to the right, but will experience bending
because of the bottom.

The bottom shape is now a function of $x$ and $y$.  An "elliptic"
Gaussian function in two dimensions, with its peak at $(B_{mx},
B_{my})$, generalizes ([2](#wave:app:exer:tsunami1D:hill:Gauss)):

$$
\begin{equation}
B =
B_0 + B_a\exp{\left(-\left(\frac{x-B_{mx}}{B_s}\right)^2
-\left(\frac{y-B_{my}}{bB_s}\right)^2\right)},
\label{wave:app:exer:tsunami2D:hill:Gauss} \tag{6}
\end{equation}
$$

where $b$ is a scaling parameter: $b=1$ gives a circular Gaussian
function with circular contour lines, while $b\neq 1$ gives an elliptic
shape with elliptic contour lines. To indicate the input parameters in the
model, we may write

$$
B = B(x;B_0,B_a,B_{mx}, B_{my} ,B_s, b)\thinspace .
$$

The "cosine hat" ([3](#wave:app:exer:tsunami1D:hill:cohat)) can also be
generalized to

$$
\begin{equation}
B =
B_0 + B_a\cos{\left( \pi\frac{x-B_{mx}}{2B_s}\right)}
\cos{\left( \pi\frac{y-B_{my}}{2B_s}\right)},
\label{wave:app:exer:tsunami2D:hill:cohat} \tag{7}
\end{equation}
$$

when $0 \leq \sqrt{x^2+y^2} \leq B_s$ and $B=B_0$ outside this circle.

A box-shaped obstacle means that

$$
\begin{equation}
B(x; B_0, B_a, B_m, B_s, b) = B_0 + B_a
\label{wave:app:exer:tsunami2D:hill:box} \tag{8}
\end{equation}
$$

for $x$ and $y$ inside a rectangle

$$
B_{mx}-B_s \leq  x \leq B_{mx} + B_s,\quad
B_{my}-bB_s \leq  y \leq B_{my} + bB_s,
$$

and $B=B_0$ outside this rectangle.
The $b$ parameter controls the rectangular shape of the cross section of
the box.

Note that the initial condition and the listed bottom shapes are
symmetric around the line $y=B_{my}$. We therefore expect the surface
elevation also to be symmetric with respect to this line.  This means
that we can halve the computational domain by working with
$[0,L_x]\times [0, B_{my}]$. Along the upper boundary, $y=B_{my}$, we
must impose the symmetry condition $\partial \eta/\partial n=0$.  Such
a symmetry condition ($-\eta_x=0$) is also needed at the $x=0$
boundary because the initial condition has a symmetry here. At the
lower boundary $y=0$ we also set a Neumann condition (which becomes
$-\eta_y=0$).  The wave motion is to be simulated until the wave hits
the reflecting boundaries where $\partial\eta/\partial n =\eta_x =0$
(one can also set $\eta =0$ - the particular condition does not matter
as long as the simulation is stopped before the wave is influenced
by the boundary condition).

Visualize the surface elevation.  Investigate how different hill shapes,
different sizes of the water gap above the hill, and different
resolutions $\Delta x = \Delta y = h$ and $\Delta t$ influence
the numerical quality of the solution.
Filename: `tsunami2D_hill`.

## Problem 7: Investigate Mayavi for visualization

Play with Mayavi code for visualizing 2D
solutions of the wave equation with variable wave velocity.
See if there
are effective ways to visualize both the solution and the wave
velocity scalar field at the same time.
Filename: `tsunami2D_hill_mlab`.

## Problem 8: Investigate visualization packages

Create some fancy 3D visualization of the water waves *and* the subsea hill
in [Problem 6: Earthquake-generated tsunami over a 3D hill](#wave:app:exer:tsunami2D:hill).
Try to make the hill transparent. Possible visualization tools are
[Mayavi](http://code.enthought.com/projects/mayavi/),
[Paraview](http://www.paraview.org/), and
[OpenDX](http://www.opendx.org/).
Filename: `tsunami2D_hill_viz`.

## Problem 9: Implement loops in compiled languages

Extend the program from [Problem 6: Earthquake-generated tsunami over a 3D hill](#wave:app:exer:tsunami2D:hill) such
that the loops over mesh points, inside the time loop, are
implemented in compiled languages.
Consider implementations in
Cython, Fortran via `f2py`, C via Cython, C via `f2py`, C/C++ via Instant,
and C/C++ via `scipy.weave`.
Perform efficiency experiments to investigate the relative performance
of the various implementations. mathcal{I}_t is often advantageous to normalize
CPU times by the fastest method on a given mesh.
Filename: `tsunami2D_hill_compiled`.

## Exercise 10: Simulate seismic waves in 2D

The goal of this exercise is to simulate seismic waves using the
PDE model ([wave:app:acoustic:seismic:accoustic1a](#wave:app:acoustic:seismic:accoustic1a)) in a
2D $xz$ domain with geological layers.
Introduce $m$ horizontal layers of thickness $h_i$, $i=0,\ldots,m-1$.
Inside layer number $i$ we have a vertical wave velocity $c_{z,i}$
and a horizontal wave velocity $c_{h,i}$. Make a program
for simulating such 2D waves. Test it on a case with 3 layers where

$$
c_{z,0}=c_{z,1}=c_{z,2},\quad c_{h,0}=c_{h,2},\quad c_{h,1} \ll c_{h,0}
\thinspace .
$$

Let $s$ be a localized point source at the middle of the Earth's surface
(the upper boundary)
and investigate how the resulting wave travels through the medium.
The source can be a localized Gaussian peak that oscillates in
time for some time interval.
Place the boundaries far enough from the expanding wave so that the
boundary conditions do not disturb the wave. Then the type of
boundary condition does not matter, except that we physically need
to have $p=p_0$, where $p_0$ is the atmospheric pressure,
at the upper boundary.
Filename: `seismic2D`.

## Project 11: Model 3D acoustic waves in a room

The equation for sound waves in air is derived in the section [wave:app:sound](#wave:app:sound)
and reads

$$
p_{tt} = c^2\nabla^2 p,
$$

where $p(x,y,z,t)$ is the pressure and $c$
is the speed of sound, taken as 340 m/s.

However, sound is absorbed in the air due to relaxation of molecules
in the gas. A model for simple relaxation, valid for gases consisting
only of one type of molecules, is a term $c^2\tau_s\nabla^2 p_t$ in
the PDE, where $\tau_s$ is the relaxation time. If we generate sound
from, e.g., a loudspeaker in the room, this sound source must also
be added to the governing equation.

The PDE with the mentioned type of damping and source then becomes

$$
\begin{equation}
p_tt = c^2\nabla^p + c^2\tau_s\nabla^2 p_t + f,
\label{_auto2} \tag{9}
\end{equation}
$$

where $f(x,y,z,t)$ is the source term.

The walls can absorb some sound. A possible model is to have a "wall layer"
(thicker than the physical wall)
outside the room where $c$ is changed such that some of the wave energy
is reflected and some is absorbed in the wall. The absorption of
energy can be taken care of by adding a damping term $bp_t$ in
the equation:

$$
\begin{equation}
p_tt + bp_t = c^2\nabla^p + c^2\tau_s\nabla^2 p_t + f\thinspace .
\label{_auto3} \tag{10}
\end{equation}
$$

Typically, $b=0$ in the room and $b>0$ in the wall. A discontinuity in $b$
or $c$ will give rise to reflections. mathcal{I}_t can be wise to use a constant
$c$ in the wall to control reflections because of the discontinuity between
$c$ in the air and in the wall, while $b$ is gradually increased
as we go into the wall to avoid reflections because of rapid changes in $b$.
At the outer boundary of the wall the condition $p=0$ or $\partial p/\partial n=0$ can be imposed. The waves should anyway be approximately dampened
to $p=0$ this far out in the wall layer.

There are two strategies for discretizing the $\nabla^2 p_t$ term: using
a center difference between times $n+1$ and $n-1$ (if the equation is
sampled at level $n$), or use a one-sided difference based on levels $n$
and $n-1$. The latter has the advantage of not leading to any equation system,
while the former is second-order accurate as the scheme for the
simple wave equation $p_tt = c^2\nabla^2 p$. To avoid an equation system,
go for the one-sided difference such that the overall scheme becomes
explicit and only of first order in time.

Develop a 3D solver for the specified PDE and introduce a wall layer.
Test the solver with the method of manufactured solutions. Make some
demonstrations where the wall reflects and absorbs the waves (reflection
because of discontinuity in $b$ and absorption because of growing $b$).
Experiment with the impact of the $\tau_s$ parameter.
Filename: `acoustics`.

## Project 12: Solve a 1D transport equation

We shall study the wave equation

$$
\begin{equation}
u_t + cu_x = 0,\quad x\in (0,L],\ t\in (0, T],
\label{wave:app:exer:advec1D:ueq} \tag{11}
\end{equation}
$$

with initial condition

$$
\begin{equation}
u(x,0) = I(x),\quad x\in [0,L],
\label{_auto4} \tag{12}
\end{equation}
$$

and *one* periodic boundary condition

$$
\begin{equation}
u(0,t) = u(L,t)
\thinspace .
\label{_auto5} \tag{13}
\end{equation}
$$

This boundary condition means that what goes out of the domain at $x=L$
comes in at $x=0$. Roughly speaking,
we need only one boundary condition because
the spatial derivative is of first order only.

**Physical interpretation.**
The parameter $c$ can be constant or variable, $c=c(x)$. The
equation ([11](#wave:app:exer:advec1D:ueq)) arises in *transport*
problems where a quantity $u$, which could be temperature or
concentration of some contaminant, is transported with the velocity
$c$ of a fluid. In addition to the transport imposed by "travelling with
the fluid", $u$ may also be transported by diffusion (such as
heat conduction or Fickian diffusion), but we have in
the model $u_t + cu_x$ assumed that diffusion effects are negligible,
which they often are.

**a)**
Show that under the assumption of $a=\hbox{const}$,

$$
\begin{equation}
u(x,t) = I(x - ct)
\label{wave:app:exer:advec1D:uexact} \tag{14}
\end{equation}
$$

fulfills the PDE as well as the initial and boundary condition
(provided $I(0)=I(L)$).

A widely used numerical scheme for ([11](#wave:app:exer:advec1D:ueq))
applies a forward difference in time and a backward difference in
space when $c>0$:

$$
\begin{equation}
[D_t^+ u + cD_x^-u = 0]_i^n
\label{wave:app:exer:advec1D:u:upwind} \tag{15}
\thinspace .
\end{equation}
$$

For $c<0$ we use a forward difference in space: $[cD_x^+u]_i^n$.

**b)**
Set up a computational algorithm and implement it in a function.
Assume $a$ is constant and positive.

**c)**
Test the implementation by using the remarkable property that
the numerical solution is exact at the mesh points if
$\Delta t = c^{-1}\Delta x$.

**d)**
Make a movie comparing the numerical and exact solution for the
following two choices of initial conditions:

$$
\begin{equation}
 I(x) = \left\lbrack\sin\left(\pi\frac{x}{L}\right)\right\rbrack^{2n}
\label{wave:app:exer:advec1D:I:sin} \tag{16}
\end{equation}
$$

where $n$ is an integer, typically $n=5$,
and

$$
\begin{equation}
I(x) = \exp{\left( -\frac{(x-L/2)^2}{2\sigma2}\right)} \thinspace .
\end{equation}
\label{wave:app:exer:advec1D:I:gauss} \tag{17}
$$

Choose $\Delta t = c^{-1}\Delta x, 0.9c^{-1}\Delta x, 0.5c^{-1}\Delta x$.

**e)**
The performance of the suggested numerical scheme can be investigated
by analyzing the numerical dispersion relation.
Analytically, we have that the *Fourier component*

$$
u(x,t) = e^{i(kx-\omega t)},
$$

is a solution of the PDE if $\omega = kc$. This is the *analytical
dispersion relation*. A complete solution of the PDE can be built by
adding up such Fourier components with different amplitudes, where the
initial condition $I$ determines the amplitudes. The solution $u$ is
then represented by a Fourier series.

A similar discrete Fourier component at $(x_p,t_n)$ is

$$
u_p^q = e^{i(kp\Delta x -\tilde\omega n\Delta t)},
$$

where in general $\tilde\omega$ is a function of $k$, $\Delta t$, and
$\Delta x$, and differs from the exact $\omega =kc$.

Insert the discrete Fourier component in the numerical scheme and
derive an expression for $\tilde\omega$, i.e.,
the discrete dispersion relation. Show in particular that if
$\Delta t/(c\Delta x)=1$, the discrete solution coincides with the
exact solution at the mesh points, regardless of the mesh resolution (!).
Show that if the stability condition

$$
\frac{\Delta t}{c\Delta x}\leq 1,
$$

the discrete Fourier component cannot grow (i.e., $\tilde\omega$ is real).

**f)**
Write a test for your implementation where you try to use information
from the numerical dispersion relation.

We shall hereafter assume that $c(x)>0$.

**g)**
Set up a computational algorithm for the variable
coefficient case and implement it in a function.
Make a test that the function works for constant $a$.

**h)**
mathcal{I}_t can be shown that
for an observer moving with velocity $c(x)$, $u$ is constant.
This can be used to derive an exact solution when $a$ varies
with $x$. Show first that

$$
\begin{equation}
u(x,t) = f(C(x) - t),
\label{wave:app:exer:advec1D:u:avar:sol} \tag{18}
\end{equation}
$$

where

$$
C'(x) = \frac{1}{c(x)},
$$

is a solution of ([11](#wave:app:exer:advec1D:ueq)) for any
differentiable function $f$.

**Solution.**
Let $\xi = C(x) - t$.
We have that

$$
u_t = f'(\xi)(-1),
$$

while

$$
u_x = f'(\xi)C'(x) = f'(\xi)\frac{1}{c(x)},
$$

implying that $au_x = f'(\xi)$. Then we have $u_t + cu_x= -f'(\xi) +
f'(\xi) = 0$.

**i)**
Use the initial condition to show that an exact solution is

$$
u(x,t) = I(C^{-1}(C(x)-t)),
$$

with $C^{-1}$ being the inverse function of $C = \int c^{1}dx$.
Since $C(x)$ is an integral $\int_0^x (1/c)dx$,
$C(x)$ is monotonically increasing and there exists hence an
inverse function $C^{-1}$ with values in $[0,L]$.

**Solution.**
In general we have $u(x,t) = f(C(x)-t)$ and the solution is
of this form with $f(\xi)=I(C^{-1}(\xi))$.
Moreover, at $t=0$ we have
$I(C^{-1}(C(x)))=I(x)$, which is the required initial condition.

To compute ([18](#wave:app:exer:advec1D:u:avar:sol)) we need to integrate
$1/c$ to obtain $C$ and then compute the inverse of $C$.

The inverse function computation can be easily done if we first think
discretely. Say we have some function $y=g(x)$ and seek its inverse.
Plotting $(x_i,y_i)$, where $y_i=g(x_i)$ for some mesh points $x_i$,
displays $g$ as a function of $x$.
The inverse function is simply $x$ as a function of $g$, i.e., the
curve with points $(y_i,x_i)$. We can therefore quickly compute points
at the curve of the inverse function. One way of extending these
points to a continuous function is to assume a linear variation
(known as linear interpolation)
between the points (which actually means to draw straight lines between
the points, exactly as done by a plotting program).

The function `wrap2callable` in `scitools.std` can take a set of points
and return a continuous function that corresponds to linear
variation between the points. The computation of the inverse of a
function $g$ on $[0,L]$ can then be done by

```
%matplotlib inline

def inverse(g, domain, resolution=101):
    x = linspace(domain[0], domain[L], resolution)
    y = g(x)
    from scitools.std import wrap2callable
    g_inverse = wrap2callable((y, x))
    return g_inverse
```

To compute $C(x)$ we need to integrate $1/c$, which can be done by
a Trapezoidal rule. Suppose we have computed $C(x_i)$ and need
to compute $C(x_{i+1})$. Using the Trapezoidal rule with $m$ subintervals
over the integration domain $[x_i,x_{i+1}]$ gives

$$
\begin{equation}
C(x_{i+1}) = C(x_i) + \int_{x_i}^{x_{i+1}} \frac{dx}{c}
\approx h\left(
\frac{1}{2}\frac{1}{c(x_i)} + \frac{1}{2}\frac{1}{c(x_{i+1})}
+ \sum_{j=1}^{m-1} \frac{1}{c(x_i + jh)}\right),
\label{wave:app:exer:advec1D:C:recursion} \tag{19}
\end{equation}
$$

where $h=(x_{i+1}-x_i)/m$ is the length of the subintervals used
for the integral over $[x_i,x_{i+1}]$.
We observe that ([19](#wave:app:exer:advec1D:C:recursion)) is a
*difference equation* which we can solve by repeatedly
applying ([19](#wave:app:exer:advec1D:C:recursion)) for
$i=0,1,\ldots,N_x-1$ if a mesh $x_0,x_,\ldots,x_{N_x}$ is prescribed.
Note that $C(0)=0$.

**j)**
Implement a function for computing $C(x_i)$ and one for computing
$C^{-1}(x)$ for any $x$. Use these two functions for computing
the exact solution $I(C^{-1}(C(x)-t))$.
End up with a function `u_exact_variable_c(x, n, c, I)` that returns
the value of $I(C^{-1}(C(x)-t_n))$.

**k)**
Make movies showing a comparison of the numerical and exact solutions
for the two initial conditions
([16](#wave:app:exer:advec1D:I:sin)) and ([17](#wave:app:exer:advec1D:I:gauss)).
Choose $\Delta t = \Delta x /\max_{0,L} c(x)$
and the velocity of the medium as

1. $c(x) = 1 + \epsilon\sin(k\pi x/L)$, $\epsilon <1$,

2. $c(x) = 1 + I(x)$, where $I$ is given by
   ([16](#wave:app:exer:advec1D:I:sin)) or ([17](#wave:app:exer:advec1D:I:gauss)).

The PDE $u_t + cu_x=0$ expresses that the initial condition $I(x)$
is transported with velocity $c(x)$.

Filename: `advec1D`.

## Problem 13: General analytical solution of a 1D damped wave equation

We consider an initial-boundary value problem for the damped wave equation:

$$
\begin{alignat*}{2}
u_{tt} +bu_t &= c^2 u_{xx}, \quad &x\in (0,L),\ t\in (0,T]\\
u(0,t) &= 0,\\
u(L,t) &=0,\\
u(x, 0) &= I(x),\\
u_t(x, 0) &= V(x)\thinspace .
\end{alignat*}
$$

Here, $b\geq 0$ and $c$ are given constants. The aim is to derive a general
analytical solution of this problem. Familiarity with the method of
separation of variables for solving PDEs will be assumed.

**a)**
Seek a solution on the form $u(x,t)=X(x)T(t)$. Insert this solution in
the PDE and show that it leads to two differential equations for $X$ and $T$:

$$
T'' + bT' + \lambda T = 0,\quad c^2 X'' +\lambda X = 0,
$$

with $X(0)=X(L)=0$ as boundary conditions, and $\lambda$ as a constant
to be determined.

**b)**
Show that $X(x)$ is on the form

$$
X_n(x) = C_n\sin kx,\quad k = \frac{n\pi}{L},\quad n=1,2,\ldots
$$

where $C_n$ is an arbitrary constant.

**c)**
Under the assumption that $(b/2)^2 < k^2$, show that $T(t)$ is on
the form

$$
T_n(t) = e^{-{\frac{1}{2}}bt}(a_n\cos\omega t + b_n\sin\omega t),
\quad\omega = \sqrt{k^2 - \frac{1}{4}b^2},\quad n=1,2,\ldots
$$

The complete solution is then

$$
u(x,t) = \sum_{n=1}^\infty \sin kx e^{-{\frac{1}{2}}bt}(
A_n\cos\omega t + B_n\sin\omega t),
$$

where the constants $A_n$ and $B_n$ must be computed from the initial
conditions.

**d)**
Derive a formula for $A_n$ from $u(x,0)=I(x)$ and developing $I(x)$ as a sine
Fourier series on $[0,L]$.

**e)**
Derive a formula for $B_n$ from $u_t(x,0)=V(x)$ and developing $V(x)$ as
a sine Fourier series on $[0,L]$.

**f)**
Calculate $A_n$ and $B_n$ from vibrations of a string where $V(x)=0$ and

$$
\begin{equation}
I(x) = \left\lbrace
\begin{array}{ll}
ax/x_0, & x < x_0,\\
a(L-x)/(L-x_0), & \hbox{otherwise}
\end{array}\right.
\label{_auto6} \tag{20}
\end{equation}
$$

**g)**
Implement a function `u_series(x, t, tol=1E-10)` for
the series for $u(x,t)$,
where `tol` is a tolerance for truncating the series. Simply sum
the terms until $|a_n|$ and $|b_b|$ both are less than `tol`.

**h)**
What will change in the derivation of the analytical solution if we
have $u_x(0,t)=u_x(L,t)=0$ as boundary conditions?
And how will you solve the problem with $u(0,t)=0$ and $u_x(L,t)=0$?

Filename: `damped_wave1D`.

## Problem 14: General analytical solution of a 2D damped wave equation

Carry out [Problem 13: General analytical solution of a 1D damped wave equation](#wave:app:exer:anal:damped:wave1D) in the 2D
case: $u_{tt}+bu_t = c^2(u_{xx}+u_{yy})$, where $(x,y)\in (0,L_x)\times
(0,L_y)$. Assume a solution on the form $u(x,y,t)=X(x)Y(y)T(t)$.
Filename: `damped_wave2D`.


```python
import numpy as np
from numpy.linalg import *
rg = matrix_rank
from IPython.display import display, Math, Latex, Markdown

from sympy import *
from sympy import Matrix, solve_linear_system
```

```python
pr = lambda s: display(Markdown('$'+str(latex(s))+'$'))
def psym_matrix(a, intro='',ending='',row=False):
    try:
        if row:
                return(intro+str(latex(a))+ending)
        else:
                display(Latex("$$"+intro+str(latex(a))+ending+"$$"))
    except TypeError:
        display(latex(a)) #TODO MAY BY...
pr = lambda s: display(Markdown('$'+str(latex(s))+'$'))

def pmatrix(a, intro='',ending='',row=False):
    if len(a.shape) > 2:
        raise ValueError('pmatrix can at most display two dimensions')
    lines = str(a).replace('[', '').replace(']', '').splitlines()
    rv = [r'\begin{pmatrix}']
    rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
    rv +=  [r'\end{pmatrix}']
    if row:
        return(intro+'\n'.join(rv)+ending)
    else:
        display(Latex('$$'+intro+'\n'.join(rv)+ending+'$$'))
def crearMatrix(name,shape=(2,2)):
    X = []
    for i in range(shape[0]):
        row = []
        for j in range(shape[1]):
            row.append(Symbol(name+'_{'+str(i*10+j+11)+'}'))
        X.append(row)
    return Matrix(X)
from re import sub
def matrix_to_word(A):
    replaced = sub(r"begin{matrix}", r"(■( ", psym_matrix(A,row=1))
    replaced = sub(r"{", r"", replaced)
    replaced = sub(r"}", r"", replaced)
    replaced = sub(r"\\\\\\\\", r" @ ", replaced)
    replaced = sub(r"\\\\", r" @ ", replaced)
    replaced = sub(r"[\[\]]", r"", replaced)
    replaced = sub(r"left", r"", replaced)
    replaced = sub(r".endmatrix.right", r" ))", replaced)
    replaced = sub(r"\\", r"", replaced)
    print(replaced )
def isNilpotent(A):
    i = 0
    for n in range(25):
        if A**n == zeros(4,4):
            return i;
        else:
            i+=1
    return 0
```

```python
D1= crearMatrix("d^1")
C1= crearMatrix("c^1")
D2= crearMatrix("d^2")
C2= crearMatrix("c^2")

psym_matrix(D1, intro="D^1=")
psym_matrix(C1, intro="C^1=")
psym_matrix(D2, intro="D^2=")
psym_matrix(C2, intro="C^2=")
```

$$D^1=\left[\begin{matrix}d^1_{11} & d^1_{12}\\d^1_{21} & d^1_{22}\end{matrix}\right]$$

$$C^1=\left[\begin{matrix}c^1_{11} & c^1_{12}\\c^1_{21} & c^1_{22}\end{matrix}\right]$$

$$D^2=\left[\begin{matrix}d^2_{11} & d^2_{12}\\d^2_{21} & d^2_{22}\end{matrix}\right]$$

$$C^2=\left[\begin{matrix}c^2_{11} & c^2_{12}\\c^2_{21} & c^2_{22}\end{matrix}\right]$$

```python
D1 = Matrix([[1,1],
             [1,2]])
C1 = Matrix([[2,1],
             [1,0]])
D2 = Matrix([[1,-1],
             [0,1]])
C2 = Matrix([[1,1],
             [0,1]])
```

```python
R1 =( Matrix([ # How???
    [ C1[0,0], 0     , C1[0,1], 0     ,],
    [ 0     , C1[0,0], 0     , C1[0,1],],
    [ C1[1,0], 0     , C1[1,1], 0     ,],
    [ 0     , C1[1,0], 0     , C1[1,1],]
])).T
T1 = Matrix([ # How???
    [D1[0,0],D1[0,1],   0  , 0    ],
    [D1[1,0],D1[1,1],   0  , 0    ],
    [ 0    , 0    ,D1[0,0],D1[0,1]],
    [ 0    , 0    ,D1[1,0],D1[1,1]]
])
R2 = (Matrix([ # How???
    [ C2[0,0], 0     , C2[0,1], 0     ,],
    [ 0     , C2[0,0], 0     , C2[0,1],],
    [ C2[1,0], 0     , C2[1,1], 0     ,],
    [ 0     , C2[1,0], 0     , C2[1,1],]
])).T
T2 = (Matrix([ # How???
    [D2[0,0],D2[0,1],   0  , 0    ],
    [D2[1,0],D2[1,1],   0  , 0    ],
    [ 0    , 0    ,D2[0,0],D2[0,1]],
    [ 0    , 0    ,D2[1,0],D2[1,1]]
]))
```

```python
psym_matrix(R1, ending=psym_matrix(T1,row=1))
psym_matrix(R2, ending=psym_matrix(T2,row=1))
```

$$\left[\begin{matrix}2 & 0 & 1 & 0\\0 & 2 & 0 & 1\\1 & 0 & 0 & 0\\0 & 1 & 0 & 0\end{matrix}\right]\left[\begin{matrix}1 & 1 & 0 & 0\\1 & 2 & 0 & 0\\0 & 0 & 1 & 1\\0 & 0 & 1 & 2\end{matrix}\right]$$

$$\left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\1 & 0 & 1 & 0\\0 & 1 & 0 & 1\end{matrix}\right]\left[\begin{matrix}1 & -1 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & -1\\0 & 0 & 0 & 1\end{matrix}\right]$$

```python
psym_matrix(R1*T1+R2*T2)
```

$$\left[\begin{matrix}3 & 1 & 1 & 1\\2 & 5 & 1 & 2\\2 & 0 & 1 & -1\\1 & 3 & 0 & 1\end{matrix}\right]$$

```python
psym_matrix(crearMatrix('g').T)
```

$$\left[\begin{matrix}g_{11} & g_{21}\\g_{12} & g_{22}\end{matrix}\right]$$

```python

```


| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

### Section18.5

```python
#
# Pierre Proulx
#
#
# Préparation de l'affichage et des outils de calcul symbolique
#
import sympy as sp
from IPython.display import *
sp.init_printing(use_latex=True)
%matplotlib inline
```

```python
# Paramètres, variables et fonctions

eta=sp.symbols('eta')
phi=sp.Function('phi')(eta)
Q=sp.Function('Q')(eta)
```

```python
#
# La solution est similaire à celle du chapitre 4, on utilisera d'abord cette solution ensuite on insèrera les
# variables voulues
#
eq1=sp.Eq(Q.diff(eta)+2*eta*Q,0)
Q=sp.dsolve(eq1).rhs
eq=sp.Eq(phi.diff(eta)-Q,0)
display(eq)
```

```python
phi=sp.dsolve(eq).rhs
display(phi)
```

```python
cl1=sp.Eq(Q.subs(eta,0),1)          # puisque en eta=0 la constante C2 disparait,
#                                     on solutionne tout de suite C1
constante1=sp.solve(cl1,sp.symbols('C1'),dict=True)
display(constante1)
```

```python
phi=phi.subs(constante1[0]) # C1 est dans une liste de 1 élément, voir ci-haut.
cl2=sp.Eq(phi.subs(eta,sp.oo),0)
display(cl2)
```

```python
constante2=sp.solve(cl2,sp.symbols('C2'),dict=True)
phi=phi.subs(constante2[0])
display(phi)
```

```python
#
#  On remplace eta et phi , 4.1-14 devient 18.5-16
#
CA,CA0=sp.symbols('C_A CA0')
x,DAB,z,Vmax=sp.symbols('x,D_AB,z,V_m')
etaCh18=x/(sp.sqrt(4*DAB*z/Vmax))
C=sp.Eq(phi.subs(eta,etaCh18),CA/CA0)
display(C)
```

```python
#
# On pourrait maintenant voir l'effet des paramètres sur le profil
# afin de comparer avec la figure 18.5-1
#
# Valeurs estimées des paramètres en utilisant la diffusivité de l'hélium dans l'eau, c'est
# un tout petit atome qui diffusera donc très rapidement
#
dico={'D_AB':7e-9,'V_m':.1}
Cplot=C.lhs.subs(dico)
delta=0.003                          # film de 3 millimètres, essayez de varier la position z pour voir
Vzplot=1-x**2/delta**2               # la combinaison profil de concentration-profil de vitesse
#                                      comme sur la figure 18.5-1.
#                                      La simplification v=v_max est-elle justifiée???
#
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize']=10,8
p=sp.plot((Cplot.subs(z,1),(x,0,delta)),(Vzplot,(x,0,delta)),show=False,legend=True)
p[0].line_color='blue'
p[0].label='concentration'
p[1].line_color='red'
p[1].label='vitesse'
p.show()
```


## Support Vector Machines:

Support vector machines focus only on the points that are the most difficult to tell apart, whereas other classifiers pay attention to all of the points.

The intuition behind the support vector machine approach is that if a classifier is good at the most challenging comparisons (the points in B and A that are closest to each other in Figure 2), then the classifier will be even better at the easy comparisons (comparing points in B and A that are far away from each other).

**Perceptrons and other classifiers:**

Perceptrons are built by taking one point at a time and adjusting the dividing line accordingly. As soon as all of the points are separated, the perceptron algorithm stops. But it could stop anywhere. Figure 1 shows that there are a bunch of different dividing lines that separate the data. The perceptron's stopping criteria is simple: "separate the points and stop improving the line when you get 100% separation". The perceptron is not explicitly told to find the best separating line. Logistic regression and linear discriminant models are built similarly to perceptrons.

The best dividing line maximizes the distance between the B points closest to A and the A points closest to B. It's not necessary to look at all of the points to do this. In fact, incorporating feedback from points that are far away can bump the line a little too far, as seen below.

**Support Vector Machines:**

Unlike other classifiers, the support vector machine is ***explicitly*** told to find the best separating line. How? The support vector machine searches for the closest points (Figure 2), which it calls the "support vectors" (the name "support vector machine" is due to the fact that points are like vectors and that the best line "depends on" or is "supported by" the closest points).

Once it has found the closest points, the SVM draws a line connecting them (see the line labeled 'w' in Figure 2). It draws this connecting line by doing vector subtraction (point A - point B). The support vector machine then declares the best separating line to be the line that bisects -- and is perpendicular to -- the connecting line.

The support vector machine is better because when you get a new sample (new points), you will have already made a line that keeps B and A as far away from each other as possible, and so it is less likely that one will spillover across the line into the other's territory.

The paper called *[Duality and Geometry in SVM Classifiers](http://www.robots.ox.ac.uk/~cvrg/bennett00duality.pdf)* finally helped me see the light; that's where I got the images from.

## Mathematical overview of how SVM's are trained and used:

## Notations ##

In the following, scalars are denoted with italic lowercases (e.g., $y,\, b$), vectors with bold lowercases (e.g., $\mathbf{w},\, \mathbf{x}$), and matrices with italic uppercases (e.g., $W$). $\mathbf{w^T}$ is the transpose of $\mathbf{w}$, and $\|\mathbf{w}\| = \mathbf{w}^T\mathbf{w}$.

Let:

- $\mathbf{x}$ be a feature vector (i.e., the input of the SVM). $\mathbf{x} \in \mathbb{R}^n$, where $n$ is the dimension of the feature vector.
- $y$ be the class (i.e., the output of the SVM). $y \in \{ -1,1\}$, i.e. the classification task is binary.
- $\mathbf{w}$ and $b$ be the parameters of the SVM: we need to learn them using the training set.
- $(\mathbf{x}^{(i)}, y^{(i)})$ be the $i^ {\text {th}}$ sample in the dataset. Let's assume we have $N$ samples in the training set.

With $n=2$, one can represent the SVM's decision boundaries as follows:

The class $y$ is determined as follows:

$$
y^{(i)}=\left\{
                \begin{array}{ll}
                  -1 &\text{ if } \mathbf{w^T}\mathbf{x}^{(i)}+b \leq -1 \\
                  1 &\text{ if } \mathbf{w^T}\mathbf{x}^{(i)}+b \ge 1 \\
                \end{array}
              \right.
$$

which can be more concisely written as $y^{(i)} (\mathbf{w^T}\mathbf{x}^{(i)}+b) \ge 1$.

## Goal  ##

The SVM aims at satisfying two requirements:

1. The SVM should maximize the distance between the two decision boundaries. Mathematically, this means we want to maximize the distance between the hyperplane defined by $\mathbf{w^T}\mathbf{x}+b = -1$ and the hyperplane defined by $\mathbf{w^T}\mathbf{x}+b = 1$. [This distance is equal to $\frac{2}{\|\mathbf{w}\|}$](https://math.stackexchange.com/a/1306417/168764). This means we want to solve $\underset{\mathbf{w}}{\operatorname{max}} \frac{2}{\|\mathbf{w}\|}$. Equivalently we want
$\underset{\mathbf{w}}{\operatorname{min}} \frac{\|\mathbf{w}\|}{2}$.

2. The SVM should also correctly classify all $\mathbf{x}^{(i)}$, which means $y^{(i)} (\mathbf{w^T}\mathbf{x}^{(i)}+b) \ge 1, \forall i \in \{1,\dots,N\}$

Which leads us to the following quadratic optimization problem:

$$\begin{align}
\min_{\mathbf{w},b}\quad &\frac{\|\mathbf{w}\|}{2}, \\
s.t.\quad&y^{(i)} (\mathbf{w^T}\mathbf{x}^{(i)}+b) \ge 1 &\forall i \in \{1,\dots,N\}
\end{align}$$

This is the **hard-margin SVM**, as this quadratic optimization problem admits a solution iff the data is linearly separable.

One can relax the constraints by introducing so-called **slack variables** $\xi^{(i)}$. Note that each sample of the training set has its own slack variable. This gives us the following quadratic optimization problem:

$$\begin{align}
\min_{\mathbf{w},b}\quad &\frac{\|\mathbf{w}\|}{2}+ C \sum_{i=1}^{N} \xi^{(i)}, \\
s.t.\quad&y^{(i)} (\mathbf{w^T}\mathbf{x}^{(i)}+b) \ge 1 - \xi^{(i)},&\forall i \in \{1,\dots,N\} \\
\quad&\xi^{(i)}\ge0, &\forall i \in \{1,\dots,N\}
\end{align}$$

This is the **soft-margin SVM**. $C$ is a hyperparameter called **penalty of the error term**.
 ([What is the influence of C in SVMs with linear kernel?](https://stats.stackexchange.com/q/31066/12359) and [Which search range for determining SVM optimal parameters?](https://stats.stackexchange.com/q/43943/12359)).

One can add even more flexibility by introducing a function $\phi$ that maps the original feature space to a higher dimensional feature space. This allows non-linear decision boundaries. The quadratic optimization problem becomes:

$$\begin{align}
\min_{\mathbf{w},b}\quad &\frac{\|\mathbf{w}\|}{2}+ C \sum_{i=1}^{N} \xi^{(i)}, \\
s.t.\quad&y^{(i)} (\mathbf{w^T}\phi \left(\mathbf{x}^{(i)}\right)+b) \ge 1 - \xi^{(i)},&\forall i \in \{1,\dots,N\} \\
\quad&\xi^{(i)}\ge0, &\forall i \in \{1,\dots,N\}
\end{align}$$

## Optimization  ##

### Solving using the Lagrangian:

### Hence, the optimization depends on the dot product of pairs of examples
--------------------------------------------------------------------------------------------
-------------------
The quadratic optimization problem can be transformed into another optimization problem named the [**Lagrangian dual problem**](https://en.wikipedia.org/w/index.php?title=Duality_(optimization)&oldid=755311099#Convex_problems) (the previous problem is called the **primal**):

$$\begin{align}
\max_{\mathbf{\alpha}}  \quad &\min_{\mathbf{w},b}  \frac{\|\mathbf{w}\|}{2}+ C \sum_{i=1}^{N} \alpha^{(i)} \left(1-\mathbf{w^T}\phi \left(\mathbf{x}^{(i)}\right)+b)\right), \\
s.t. \quad&0 \leq  \alpha^{(i)} \leq C, &\forall i \in \{1,\dots,N\}
\end{align}$$

This optimization problem can be simplified (by setting some gradients to $0$) to:

$$\begin{align}
\max_{\mathbf{\alpha}}  \quad & \sum_{i=1}^{N} \alpha^{(i)} - \sum_{i=1}^{N}\sum_{j=1}^{N}  \left( y^{(i)}\alpha^{(i)}\phi\left(\mathbf{x}^{(i)}\right)^T \phi\left(\mathbf{x}^{(j)}\right) y^{(j)}\alpha^{(j)} \right), \\
s.t. \quad&0 \leq  \alpha^{(i)} \leq C, &\forall i \in \{1,\dots,N\}
\end{align}$$

$\mathbf{w}$ doesn't appear as $\mathbf{w}=\sum_{i =1}^{N}\alpha^{(i)}y^{(i)}\phi\left(x^{(i)}\right)$ (as stated by the [representer theorem](https://en.wikipedia.org/wiki/Representer_theorem)).

We therefore learn the $\alpha^{(i)}$ using the $(\mathbf{x}^{(i)}, y^{(i)})$ of the training set.

(FYI: [Why bother with the dual problem when fitting SVM?](https://stats.stackexchange.com/q/19181/12359) short answer: faster computation + allows to use the kernel trick, though there exist some good methods to train SVM in the primal e.g. see  {1})

## Making a prediction  ##

Once the $\alpha^{(i)}$ are learned, one can predict the class of a new sample with the feature vector $\mathbf{x}^{\text {test}}$ as follows:

\begin{align*}
y^{\text {test}}&=\text {sign}\left(\mathbf{w^T}\phi\left(\mathbf{x}^{\text {test}}\right)+b\right) \\
&= \text {sign}\left(\sum_{i =1}^{N}\alpha^{(i)}y^{(i)}\phi\left(x^{(i)}\right)^T\phi\left(\mathbf{x}^{\text {test}}\right)+b \right)
\end{align*}

The summation $\sum_{i =1}^{N}$ could seem overwhelming, since it means one has to sum over all the training samples, but the vast majority of $\alpha^{(i)}$ are $0$ (see [Why are the Lagrange multipliers sparse for SVMs?](https://stats.stackexchange.com/q/54976/12359)) so in practice it isn't an issue. (note that [one can construct special cases where all $\alpha^{{(i)}} > 0$](https://stats.stackexchange.com/q/110598/12359).) $\alpha^{{(i)}}=0$ iff $x^{{(i)}}$ is a **support vector**. The illustration above has 3 support vectors.

# Some comments on Duality and KTT conditions
### Primal problem

Picking up from @Antoni's post in between equations $(4)$ and $(5)$, recall that our original, or **primal**, optimization problem is of the form:

\begin{aligned}
	\min_{w, b} f(w,b) & = \min_{w, b}  \  \frac{1}{2} ||w||^2
	\\
	s.t. \ \  g_i(w,b) &= - y^{(i)} (w^T x^{(i)} + b) + 1 = 0
\end{aligned}

### Lagrange method
The method of Lagrange multipliers allows us to turn a constrained optimization problem into an unconstrained one of the form:

$$\mathcal{L}(w, b, \alpha) =   \frac{1}{2} ||w||^2 - \sum_i^m \alpha_i [y^{(i)} (w^T x^{(i)} + b) - 1]$$

Where  $\mathcal{L}(w, b, \alpha)$ is called the **Lagrangian** and $\alpha_i$ are called the **Lagrangian multipliers**.

Our **primal** optimization problem with the Lagrangian becomes the following: (note that the use of $min$, $max$ is not the most rigorous as we should also be using $\inf$ and $\sup$ here...)

$$ \min_{w,b} \left( \max_\alpha \mathcal{L}(w, b, \alpha)\right)$$

### Dual problem
What @Antoni and Prof. Patrick Winston have done in their derivation is assume that the optimization function and the constraints meet some technical conditions such that we can do the following:

$$ \min_{w,b} \left( \max_\alpha \mathcal{L}(w, b, \alpha)\right) =   \max_\alpha \left( \min_{w,b} \mathcal{L}(w, b, \alpha)\right)$$

This allows us to take the partial derivatives of $\mathcal{L}(w, b, \alpha)$ with respect to $w$ and $b$, equate to zero and then plug the results back into the original equation of the Lagrangian, hence generating an equivalent **dual** optimization problem of the form

\begin{aligned}
	&\max_{\alpha} \min_{w,b} \mathcal{L}(w,b,\alpha)
	\\
	& \max_{\alpha} \sum_i^m \alpha_i - \frac{1}{2} \sum_{i,j}^m y^{(i)}y^{(j)} \alpha_i \alpha_j <x^{(i)} x^{(j)}>
	\\
	& s.t. \ \alpha_i \geq 0
	\\
	& s.t. \ \sum_i^m \alpha_i y^{(i)} = 0
\end{aligned}

### Duality and KTT
Without going into excessive mathematical technicalities, these conditions are a combination of the Duality and the Karush Kuhn Tucker (KTT) conditions and allow us to solve the **dual** problem instead of the **primal** one, while ensuring that the optimal solution is the same. In our case the conditions are the following:

- The primal objective and inequality constraint functions must be convex
- The equality constraint function must be affine
- The constraints must be strictly feasible

Then there exists $w^*, \alpha^*$ which are solutions to the primal and dual problems. Moreover, the parameters $w^*, \alpha^*$ satisfy the KTT conditions below:

\begin{aligned}
	&\frac{\partial}{\partial w_i}  \mathcal{L}(w^*, \alpha^*, \beta^*) = 0 &(A)
	\\
	&\frac{\partial}{\partial \beta_i}  \mathcal{L}(w^*, \alpha^*, \beta^*) = 0 &(B)
	\\
	&\alpha_i^* g_i(w^*) = 0 &(C)
	\\
	&g_i(w^*) \leq 0  &(D)
	\\
	&\alpha_i^* \geq 0 &(E)
\end{aligned}

Moreover, if some $w^*, \alpha^*$ satisfy the KTT solutions then they are also solution to the primal and dual problem.

Equation $(C)$ above is of particular importance and is called the *dual complementarity condition*. It implies that if $\alpha_i^* > 0$ then $g_i(w^*) = 0$ which means that the constraint $g_i(w) \leq 0$ is active, i.e. it holds with equality rather than inequality. This is the explanation behind equation $(2)$ in Antoni's derivation where the inequality constraint is turned into an equality constraint.

### A intuitive but informal diagram

----------
---------
## Kernel trick  ##

One can observe that the optimization problem uses the $\phi\left(\mathbf{x}^{(i)}\right)$ only in the inner product $\phi\left(\mathbf{x}^{(i)}\right)^T \phi\left(\mathbf{x}^{(j)}\right)$. The function that maps $\left(\mathbf{x}^{(i)},\mathbf{x}^{(j)}\right)$ to the inner product $\phi\left(\mathbf{x}^{(i)}\right)^T \phi\left(\mathbf{x}^{(j)}\right)$ is [called](https://stats.stackexchange.com/a/226196/12359) a **kernel**, a.k.a. kernel function, often denoted by $k$.

One can choose $k$ so that the inner product is efficient to compute. This allows to use a potentially high feature space at a low computational cost. That is called the [**kernel trick**](https://en.wikipedia.org/w/index.php?title=Kernel_method&oldid=747269865#Mathematics:_the_kernel_trick). For a kernel function to be *valid*, i.e. usable with the kernel trick, it should satisfy [two key properties](https://stats.stackexchange.com/q/48506/12359). There exist [many kernel functions to choose from](https://stats.stackexchange.com/q/18030/12359). As a side note, the kernel trick [may be applied to other machine learning models](https://stats.stackexchange.com/q/2167/12359), in which case they are referred as [*kernelized*](https://scholar.google.com/scholar?q=kernelized+&hl=en&as_sdt=0,22).

## Going further  ##

Some interesting QAs on SVMs:

- [Best way to perform multiclass SVM](https://stats.stackexchange.com/q/21465/12359)
- [Support vector machines and regression](https://stats.stackexchange.com/q/13194/12359)
- [Understanding the different formulations for SVM](https://stats.stackexchange.com/q/157827/12359)
- [What's the difference between $\ell_1$-SVM, $\ell_2$-SVM and LS-SVM loss functions?](https://stats.stackexchange.com/q/255486/12359)
- To deal with an unbalanced dataset:
 - [Best way to handle unbalanced multiclass dataset with SVM](https://stats.stackexchange.com/q/20948/12359)
 - [A priori selection of SVM class weights](https://stats.stackexchange.com/a/24969/12359)
- [How does one interpret SVM feature weights?](https://stats.stackexchange.com/q/39243/12359)
- [Interpretating the C value in a linear SVM](https://stats.stackexchange.com/q/108639/12359)
- [Generalization bounds on SVM](https://stats.stackexchange.com/q/259788/12359)
- [General formula for the VC Dimension of a SVM](https://stats.stackexchange.com/q/255301/12359)
- [What does the "machine" in "support vector machine" and "restricted Boltzmann machine" mean?](https://stats.stackexchange.com/q/261041/12359)
- [How are SVMs = Template Matching?](https://stats.stackexchange.com/q/263587/12359)
- [Single layer NeuralNetwork with ReLU activation equal to SVM?](https://stats.stackexchange.com/q/190883/12359)
- [Comparing SVM and logistic regression](https://stats.stackexchange.com/q/95340/12359)

Other links:

-  [Least squares support vector machine](https://en.wikipedia.org/wiki/Least_squares_support_vector_machine)

----------
References:

- {1} Chapelle, Olivier. "Training a support vector machine in the primal." Neural computation 19, no. 5 (2007): 1155-1178. https://scholar.google.com/scholar?cluster=469291847682573606&hl=en&as_sdt=0,22 ; http://www.chapelle.cc/olivier/pub/neco07.pdf

## Code sample 1: SVM using cvxopt

```python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import pickle
from cvxopt import matrix, solvers
```

```python
# Generating the data

DIM = 2
COLORS = ['red', 'blue']

# 2-D mean of ones
M1 = np.ones(DIM, )

# 2-D mean of threes
M2 = 3*np.ones(DIM, )

# 2-D covariance of 0.3
C1 = np.diag(0.3*np.ones(DIM, ))

# 2-D covariance of 0.2
C2 = np.diag(0.2*np.ones(DIM, ))

def generate_gaussian(m , c, num):
    return np.random.multivariate_normal(m, c, num)

def plot_data_with_labels(x, y):
    unique = np.unique(y)
    for li in range(len(unique)):
        x_sub = x[y == unique[li]]
        plt.scatter(x_sub[:, 0], x_sub[:, 1], c = COLORS[li])
        plt.title('Data generated from 2 gaussians')
    plt.show()

NUM = 50

if __name__ == '__main__':
    # generate 50 points from gaussian 1
    x1 = generate_gaussian(M1, C1, NUM)

    # labels
    y1 = np.ones(x1.shape[0], )

    # generate 50 points from gaussian 2
    x2 = generate_gaussian(M2, C2, NUM)
    y2 = -np.ones(x2.shape[0], )

    # join
    x = np.concatenate((x1, x2), axis = 0)
    y = np.concatenate((y1, y2), axis = 0)

    print(f" x: {x.shape} and y: {y.shape}")

    plot_data_with_labels(x, y)

    with open('gaussiandata.pickle', 'wb') as f:
        pickle.dump((x,y), f)
```

### Fitting an SVM

```python
def fit(x, y):
    NUM = x.shape[0]
    DIM = x.shape[1]
    # we'll solve the dual
    # obtain the kernel
    K = y[:, None] * x
    K = np.dot(K, K.T)
    P = matrix(K)
    q = matrix(-np.ones((NUM, 1)))
    G = matrix(-np.eye(NUM))
    h = matrix(np.zeros(NUM))
    A = matrix(y.reshape(1, -1))
    b = matrix(np.zeros(1))
    solvers.options['show_progress'] = False
    sol = solvers.qp(P, q, G, h, A, b)
    alphas = np.array(sol['x'])
    return alphas
```

Note that `sol['x']` contains the x that was part of cvxopt interface - it is our α. Now if you are familiar with SVMs, you will know that only a few of the alphas should be non-zero and they will be our support vectors. Using these alphas, we can obtain w and b from our original SVM problem. Once we do that, they will together define the decision boundary. w is equal to Σiαiyixi and b is equal to yi−wTxi for any i such that αi>0. So we do the following to obtain them.

```python
with open('gaussiandata.pickle', 'rb') as f:
    data = pickle.load(f)

x, y = data[0], data[1]

# fit svm classifier

alphas = fit(x, y)

# get weights
w = np.sum(alphas * y[:, None] * x, axis = 0)

# get bias
cond = (alphas > 1e-4).reshape(-1)
b = y[cond] - np.dot(x[cond], w)
bias = b[0]
```

Finally, we’ll plot the decision boundary for good visualizaiton. Since it will be a line in this case, we need to obtain the slope and intercept of the line from the weights and bias.

```python
def plot_data_with_labels(x, y, ax):
    unique = np.unique(y)
    for li in range(len(unique)):
        x_sub = x[y == unique[li]]
        ax.scatter(x_sub[:, 0], x_sub[:, 1], c = COLORS[li])

def plot_separator(ax, w, b):
    slope = -w[0] / w[1]
    intercept = -b / w[1]
    x = np.arange(0, 6)
    ax.plot(x, x * slope + intercept, 'k-')

# # normalize
# norm = np.linalg.norm(w)
# w, bias = w / norm, bias / norm

fig, ax = plt.subplots()
plot_separator(ax, w, bias)
plot_data_with_labels(x, y, ax)
plt.show()
```

## Code sample 2: SVM using cvxopt

**We introduce an additional variable stickily for convenience. The variable y will be equal to positive one for all positive samples and negative one for all negative samples.**

**Next, we need to address the process by which we go about maximizing the margin. To get an equation for the width of the margin, we subtract the first support vector from the one below it and the multiply the result by the unit vector of w which is always perpendicular to the decision boundary.**

**Using the constraints from above and a bit of algebra, we get the following.**

**Therefore, in order to select the optimal decision boundary, we must maximize the equation we just computed.**

**Now, in most machine learning algorithms, we’d use something like gradient descent to minimize said function, however, for support vector machines, we use the Lagrangian. In essence, using Lagrangian, we can solve for the global minimum like we’d do in high school level calculus (i.e. take the derivative of the function and make it equal to zero). The Lagrange tells us to subtract the cost function by the summation over all the constraints where each of those constraints will be multiplied by some constant alpha (normally written as lambda for the Lagrangian).**

**Then, we perform some more algebra, plugging the equations we found in the previous step back into the original equation.**

**Before we can proceed any further, we need to express the equation in terms of matrices instead of summations. The reason being, the** `qp` **function from the CVXOPT library, which we’ll use to solve the Lagrangian, accepts very specific arguments. Thus, we need to go from:**

```python
import numpy as np
import cvxopt
from sklearn.datasets.samples_generator import make_blobs
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC
from matplotlib import pyplot as plt
```

    /home/abhishek/anaconda3/envs/tf2/lib/python3.7/site-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.datasets.samples_generator module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.datasets. Anything that cannot be imported from sklearn.datasets is now part of the private API.
      warnings.warn(message, FutureWarning)

```python
class SVM:
    def fit(self, X, y):
        n_samples, n_features = X.shape

        # P = X^T.X
        K = np.zeros((n_samples, n_samples))
        for i in range(n_samples):
            for j in range(n_samples):
                K[i,j] = np.dot(X[i], X[j])

        P = cvxopt.matrix(np.outer(y,y) * K)

        # q = -1 (1xN)
        q = cvxopt.matrix(np.ones(n_samples) * -1)

        # A = y^T
        A = cvxopt.matrix(y, (1, n_samples))

        # b=0
        b = cvxopt.matrix(0.0)

        # G = -1(NxN)
        G = cvxopt.matrix(np.diag(np.ones(n_samples) * -1))

        # h = 0 (1xN)
        h = cvxopt.matrix(np.zeros(n_samples))

        solution = cvxopt.solvers.qp(P, q, G, h, A, b)

        # Lagrange multipliers
        a = np.ravel(solution['x'])

        # Lagrange have non zero lagrange multipliers
        sv = a > 1e-5
        ind = np.arange(len(a))[sv]
        self.a = a[sv]
        self.sv = X[sv]
        self.sv_y = y[sv]

        # Intercept
        self.b = 0
        for n in range(len(self.a)):
            self.b += self.sv_y[n]
            self.b -= np.sum(self.a * self.sv_y * K[ind[n], sv])
        self.b /= len(self.a)

        # Weights
        self.w = np.zeros(n_features)
        for n in range(len(self.a)):
            self.w += self.a[n] * self.sv_y[n] * self.sv[n]

    def project(self, X):
        return np.dot(X, self.w) + self.b

    def predict(self, X):
        return np.sign(self.project(X))
```

To keep things simple, we’ll use the scikit-learn library to generate linearly separable data. We label the negative samples as -1 instead of 0. cvxopt expects the data to be in a specific format which is why we take an intermediate step.

```python
X, y = make_blobs(n_samples=250, centers=2,
                  random_state=0, cluster_std=0.60)
y[y == 0] = -1

tmp = np.ones(len(X))
y = tmp * y
```

```python
plt.scatter(X[:, 0], X[:, 1], c=y, cmap='winter')
```

```python
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
```

```python
svm = SVM()
svm.fit(X_train, y_train)
```

         pcost       dcost       gap    pres   dres
     0: -1.8226e+01 -3.4458e+01  6e+02  2e+01  2e+00
     1: -2.5252e+01 -1.8773e+01  2e+02  9e+00  7e-01
     2: -5.3459e+01 -3.2711e+01  2e+02  7e+00  6e-01
     3: -7.8360e+01 -2.6482e+01  1e+02  4e+00  3e-01
     4: -5.6818e+00 -5.1750e+00  1e+01  2e-01  1e-02
     5: -3.6906e+00 -4.1082e+00  4e-01  1e-15  8e-15
     6: -4.0061e+00 -4.0104e+00  4e-03  1e-15  6e-15
     7: -4.0094e+00 -4.0094e+00  4e-05  2e-15  7e-15
     8: -4.0094e+00 -4.0094e+00  4e-07  1e-17  5e-15
    Optimal solution found.

```python
def f(x, w, b, c=0):
    return (-w[0] * x - b + c) / w[1]

plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap='winter')

# w.x + b = 0
a0 = -4; a1 = f(a0, svm.w, svm.b)
b0 = 4; b1 = f(b0, svm.w, svm.b)
plt.plot([a0,b0], [a1,b1], 'k')

# w.x + b = 1
a0 = -4; a1 = f(a0, svm.w, svm.b, 1)
b0 = 4; b1 = f(b0, svm.w, svm.b, 1)
plt.plot([a0,b0], [a1,b1], 'k--')

# w.x + b = -1
a0 = -4; a1 = f(a0, svm.w, svm.b, -1)
b0 = 4; b1 = f(b0, svm.w, svm.b, -1)
plt.plot([a0,b0], [a1,b1], 'k--')
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

### REFERENCES:
- [In depth SVM](https://jakevdp.github.io/PythonDataScienceHandbook/05.07-support-vector-machines.html)
- [SVM example](https://towardsdatascience.com/support-vector-machine-python-example-d67d9b63f1c8)
- [SVM python](https://towardsdatascience.com/support-vector-machine-python-example-d67d9b63f1c8)
- [Support Vector Machine: Python implementation using CVXOPT](https://xavierbourretsicotte.github.io/SVM_implementation.html)
- [SVM CVXOPT](http://goelhardik.github.io/2016/11/28/svm-cvxopt/)
- [How does a Support Vector Machine (SVM) work?](https://stats.stackexchange.com/questions/23391/how-does-a-support-vector-machine-svm-work/353605#353605)
- [Help me understand Support Vector Machines](https://stats.stackexchange.com/questions/3947/help-me-understand-support-vector-machines)
- [MIT Prof. Patrick Winston](https://www.youtube.com/watch?v=_PwhiWxHK8o&feature=youtu.be)
- [How can SVM 'find' an infinite feature space where linear separation is always possible?](https://stats.stackexchange.com/questions/80398/how-can-svm-find-an-infinite-feature-space-where-linear-separation-is-always-p/168309#168309)
- [Khan academy Lagrange multiplier](https://www.youtube.com/watch?v=hQ4UNu1P2kw)
- [Support Vector Machine — Using Numpy](https://medium.com/@saishruthi.tn/support-vector-machine-using-numpy-846f83f4183d)
- [Mathematics behind SVM(Support Vector Machine)](https://towardsdatascience.com/mathematics-behind-svm-support-vector-machines-84742ddda0ca)
- [Math behind SVM (Support Vector Machine)](https://medium.com/@ankitnitjsr13/math-behind-support-vector-machine-svm-5e7376d0ee4d)
- [SVM From Scratch — Python](https://towardsdatascience.com/svm-implementation-from-scratch-python-2db2fc52e5c2)

```python

```

ML-practice/Supervised Learning/Support Vector Machines (SVM)/Support Vector Machines.ipynb