# 02 Data Structures and Libraries

# Review Exercises - Solutions

## Review Exercise 1 : Combining Imported Functions

The dot product of two vectors can be found as:

\begin{align}
\mathbf{A} \cdot \mathbf{B} = |\mathbf{A}| |\mathbf{B}| cos(\theta)
\end{align}

Where:

$\theta$ is the angle between the two vectors

$|\mathbf{A}|$ is the magnitude of vector $\mathbf{A}$.

$|\mathbf{B}|$ is the magnitude of vector $\mathbf{B}$.

The magnitude of an $n$-length vector $ \mathbf{A} = [A_1, ..., A_n]$ is:

$|\mathbf{A}| = \sqrt{A_1^2 + ... + A_n^2}$

Find the angle between the vectors `a` and `b`.

*Hint:*

Use a numpy function from this class to find the dot product.

Search online to find a numpy function that computes *magnitude*.

Search online to find a numpy function for the *inverse cosine*.

```python
# Review Exercise 1 : Find the angle between a and b
a = [9, 2, 7]
b = [4, 8, 10]
```

```python
# Review Exercise 1 : Find the angle between a and b
# Example Solution
import numpy as np

a = [9, 2, 7]
b = [4, 8, 10]

ab = np.dot(a, b)
maga = np.linalg.norm(a)
magb = np.linalg.norm(b)

theta =  np.arccos(ab / (maga * magb))

print(theta)
```

    0.6672196386878

## Review Exercise 2 :  Classifer

The dot product also indicates if the angle between two vectors $\mathbf{A}$ and $\mathbf{B}$ is:

   - acute ($\mathbf{A} \cdot \mathbf{B}>0$)
   - obtuse ($\mathbf{A} \cdot \mathbf{B}<0$)
   - right angle ($\mathbf{A} \cdot \mathbf{B}==0$)

Using `if`, `elif` and `else`, classify the angle between `a` and `b` as acute, obtuse or right angle.

```python
# Review Exercise 2 :  Classifer

a = [-1, 2, 6]
b = [4, 3, 3]

```

```python
# Review Exercise 2 :  Classifer
# Example Solution

a = [-1, 2, 6]
b = [4, 3, 3]

ab = np.dot(a, b)

if ab > 0:
    print("theta is acute")
elif ab < 0:
    print("theta is obtuse")
else:
    print("theta is right angle")
```

    theta is acute

## Review Exercise 3: Numpy Package Functions.
Find a function in the Python Numpy documentation that matches the function definition and use it to solve the problems below:

__(A)__ Definition: *Calculates the exponential function, $y= e^x$ for all elements in the input array.*

Print a list where each element is the exponential function of the corresponding element in list `a = [0.1, 0, 10]`

```python
# Review Exercise 3A
# Print a list where each element is the exponential function of the corresponding element in list a
```

```python
# Review Exercise 3A
# Example Solution
a = [0.1, 0, 10]
print(np.exp(a))
```

    [1.10517092e+00 1.00000000e+00 2.20264658e+04]

__(B)__ Definition: *Converts angles from degrees to radians.*

Convert angle `theta`, expressed in degrees, to radians:

`theta` = 47

```python
# Review Exercise 3B
# convert angle `theta`, expressed in degrees, to radians
```

```python
# Review Exercise 3B
# Example Solution

np.radians(47)
```

    0.8203047484373349

__(C)__ Definition: *Return the positive square-root of an array, element-wise.*

Generate an array where each element is the square root of the corresponding element in array `a = ([4, 16, 81])`

```python
# Review Exercise 3C
# Print a list where each element is the square root of the corresponding element in list a
```

```python
# Review Exercise 3C
# Example Solution

a = ([4, 16, 81])
print(np.sqrt(a))
```

    [2. 4. 9.]

## Review Exercise 4:  Using a single list with a `for` loop.
In the cell below, use a `for` loop to print the first letter of each month in the list.

```python
# Review Exercise 4
# Print the first letter of each month in the list

months = ["January",
         "February",
         "March",
         "April",
         "May",
         "June",
         "July",
         "August",
         "September",
         "October",
         "November",
         "December"]
```

```python
# Review Exercise 4
# Example Solution
months = ["January",
         "February",
         "March",
         "April",
         "May",
         "June",
         "July",
         "August",
         "September",
         "October",
         "November",
         "December"]

for m in months:
    print(m[0], end="")
```

    JFMAMJJASOND


# Transforming distributions with Normalizing Flows

Probability distributions are all over machine learning. They can determine the structure of a model for supervised learning (are we doing linear regression over a Gaussian random variable, or is it categorical?); and they can serve as goals in unsupervised learning, to train generative models that we can use to evaluate the probability density for any observation, or to generate novel samples.

Normalizing Flows are a particular mechanism for dealing with probability distributions, that continues to receive increased attention from the machine learning community. Just a few weeks ago, a whole workshop was devoted to them (and some related methods) at the ICML 2020 conference. So what are Normalizing Flows and why should we care about them?

In this post I will attempt to answer this question, covering

- An introductory description of NFs and their applications
- A basic implementation using PyTorch
- The results after training a model for density estimation with NFs

This notebook runs in under 20 seconds in CPU, on a laptop with an Intel Core i7 processor, so I'd highly encourage you to run it and tinker with it.

---

Normalizing flows are based on a fundamental result from probability theory. Assume $\textbf{u}$ is a vector in a d-dimensional space $\mathbb{R}^D$, obtained by sampling a random variable with probability density $p_u(\mathbf{u})$. A normalizing flow is a differentiable transformation $T$ with inverse $T^{-1}$, such that if we pass $\mathbf{u}$ through $T$, we get another vector $T(\mathbf{u}) = \mathbf{x}$ in $\mathbb{R}^D$. Since $\mathbf{u}$ is a sample from a random variable, it follows that $\mathbf{x}$ is also a sample from another random variable. It turns out that the density of the transformed random variable can be computed as

$$
p_x(\mathbf{x}) = p_u(\mathbf{u})\left\vert \text{det } \mathbf{J}_T(\mathbf{u}) \right\vert^{-1}\tag{1}
$$

In this formula, $\mathbf{J}_T(\mathbf{u})$ is the Jacobian of the transformation $T$ with respect to $\mathbf{u}$. This is a $D\times D$ matrix that contains all partial derivatives of the elements in $\mathbf{x}$ with respect to all elements in $\mathbf{u}$:
$$
\mathbf{J}_T(\mathbf{u}) =
\begin{bmatrix}
\frac{\partial T_1}{\partial u_1} & \dots & \frac{\partial T_1}{\partial u_D} \\
\vdots & \ddots & \vdots \\
\frac{\partial T_D}{\partial u_1} & \dots & \frac{\partial T_d}{\partial u_D}
\end{bmatrix}
$$

The above fact is particularly interesting if $p_u(\mathbf{u})$ is a distribution for which we can easily evaluate the density (such as a Gaussian distribution), and $T$ is a flexible transformation that can potentially produce a more complicated density $p_x(\mathbf{x})$:

What is interesting is that we don't need explicit access to the transformed density to evaluate it for any $\mathbf{x}$. Instead, following eq. 1, we can transform $\mathbf{x}$ back to $T^{-1}(\mathbf{x}) = \mathbf{u}$, evaluate the density with $p_u(\mathbf{u})$, and multiply the result by the Jacobian of the inverse:
$$
\begin{align}
p_x(\mathbf{x}) &= p_u(\mathbf{u})\left\vert \text{det } \mathbf{J}_T(\mathbf{u}) \right\vert^{-1} \\
&= p_u(T^{-1}(\mathbf{x}))\left\vert \text{det } \mathbf{J}_{T^{-1}}(\mathbf{x}) \right\vert
\end{align}
$$

In fact, we can specify $T$ as a composition of $K$ differentiable, invertible transformations $T = T_K \circ\dots\circ T_1$, to obtain a more flexible flow, and the density is simply modified with products of determinants:
$$
p_x(\mathbf{x}) = p_u(T^{-1}(\mathbf{x}))\prod_{k = 1}^K\left\vert \text{det } \mathbf{J}_{T_k^{-1}}(\mathbf{x}) \right\vert
$$

This nice property has been exploited for density estimation in high dimensional spaces with complicated distributions, such as images. By mapping them back to a simple distribution using a normalizing flow, we can optimize the parameters $\theta$ of a generative model by maximizing its log-likelihood $\log p_x(\mathbf{x};\theta)$. In particular, for a dataset with $N$ observations, we want to maximize
$$
\log \prod_{i=1}^N p(\mathbf{x}^{(i)};\theta) = \sum_{i = 1}^N\log p_u(T^{-1}(\mathbf{x}^{(i)});\theta) + \sum_{k=1}^K\log\vert\text{det }\mathbf{J}_{T_k^{-1}}(\mathbf{x}^{(i)}; \theta)\vert\tag{2}
$$
Note that the parameters of the model can be the parameters of the base density $p_u(\mathbf{u})$ (such as the mean and covariance), as well as any parameters involved in the transformation $T$.

Once we have trained a model with normalizing flows, we have a generative model that we can use to

- Evaluate the density $p_x(\mathbf{x})$ for a new observation
- Generate samples that look like observations used for training, by sampling $\mathbf{u}\sim p_u(\mathbf{u})$  and passing it through the flow to get $\mathbf{x} = T(\mathbf{u})$.

There is one last caveat to mention before we can actually use normalizing flows with neural networks: the determinant of the Jacobian. In general, computing the determinant of a $D\times D$ requires a computation proportional to $D^3$, which can become unfeasible for large $D$. There is a special case, however. For a *triangular* matrix, the determinant reduces to the product of its elements in the diagonal. This then turns the amount of computation proportional to $D$. The reduction from cubic to linear complexity is a key motivator for the use of normalizing flows in machine learning.

## Implementing tractable flows

One way to implement a normalizing flow that results in a triangular Jacobian is to use *coupling layers*, introduced by Dihn et al. (2015). A coupling layer leaves the first $d$ elements unchanged, and the rest of the elements are linearly transformed with a (possibly nonlinear) function of the first $d$ elements. More formally, if $\mathbf{z}$ is the input vector and by $\mathbf{z}_{1:d}$ we denote a slice of the first $d$ elements, a coupling layer implements the following equations to produce the output vector $\mathbf{z}'$:
$$
\begin{align}
\mathbf{z}'_{1:d} &= \mathbf{z}_{1:d} \\
\mathbf{z}'_{d+1:D} &= \mathbf{z}_{d+1:D}\odot\exp(s(\mathbf{z}_{1:d})) + t(\mathbf{z}_{1:d})
\end{align}
$$
where $\odot$ denotes element-wise multiplication, and $s$ and $t$ are scaling and translation functions that can be implemented with neural networks.

We may now ask two questions:

1) What is the inverse of this transformation? We can easily invert the two equations above to obtain
$$
\begin{align}
\mathbf{z}_{1:d} &= \mathbf{z}_{1:d}' \\
\mathbf{z}_{d+1:D} &= (\mathbf{z}_{d+1:D}' - t(\mathbf{z}_{1:d}'))\odot\exp(-s(\mathbf{z}_{1:d}'))
\end{align}
$$

2) What is the determinant of the Jacobian? Since the first $d$ elements of the output are the same as the input, there will be a $d\times d$ block in the Jacobian containing an identity matrix. The remaining elements only depend on the first $d$, and themselves, so the Jacobian is triangular with the following structure:
$$
\mathbf{J}(\mathbf{z}) = \begin{bmatrix}
\mathbf{I} & \mathbf{0} \\
\mathbf{A} & \text{diag}(\exp(s(\mathbf{z}_{1:d})))
\end{bmatrix}
$$

In this case, $\mathbf{A}$ is a matrix of derivatives we don't care about, and the determinant reduces to the products of the diagonal in the matrix $\text{diag}(\exp(s(\mathbf{z}_{1:d})))$, that simplifies further when computing the logarithm:
$$
\begin{align}
\log\vert\text{det }\mathbf{J}(\mathbf{z})\vert &= \log\prod_{i=1}^d \exp(s(\mathbf{z}_{1:d})_i) \\
&= \sum_{i=1}^d s(\mathbf{z}_{1:d})_i
\end{align}
$$

Lastly, for the inverse of the transformation we have
$$
\log\vert\det\mathbf{J}(\mathbf{z})\vert^{-1} = -\sum_{i=1}^d s(\mathbf{z}_{1:d})_i
$$
We now have all the ingredients to implement a coupling layer. To select only some parts of the input we will use a binary tensor to mask values, and for the scaling and translation functions we will use a 2-layer MLP, sharing parameters for both functions.

```python
import torch.nn as nn

class Coupling(nn.Module):
    def __init__(self, dim, num_hidden, mask):
        super().__init__()

        # The MLP for the scaling and translation functions
        self.nn = torch.nn.Sequential(nn.Linear(dim // 2, num_hidden),
                                      nn.ReLU(),
                                      nn.Linear(num_hidden, num_hidden),
                                      nn.ReLU(),
                                      nn.Linear(num_hidden, dim))

        # Initialize the coupling to implement the identity transformation
        self.nn[-1].weight.data.zero_()
        self.nn[-1].bias.data.zero_()

        self.register_buffer('mask', mask)

    def forward(self, z, log_det, inverse=False):
        mask = self.mask
        neg_mask = ~mask

        # Compute scale and translation for relevant inputs
        s_t = self.nn(z[:, mask])
        s, t = torch.chunk(s_t, chunks=2, dim=1)

        if not inverse:
            z[:, neg_mask] = z[:, neg_mask] * torch.exp(s) + t
            log_det = log_det + torch.sum(s, dim=1)
        else:
            z[:, neg_mask] = (z[:, neg_mask] - t) * torch.exp(-s)
            log_det = log_det - torch.sum(s, dim=1)

        return z, log_det
```

Now that we have defined a module for a coupling layer, we can use composition of coupling layers to build a normalizing flow.

We will use the `forward` method of the flow to compute the density of an observation $\mathbf{x}$ following equation 2. Additionally, we will define a `sample` method that takes a sample from the base distribution $p_u(\mathbf{u})$ and passes it through the flow in the inverse direction to obtain what would be a sample from the target distribution. These two methods traverse the flow in one or another direction, which we define in the `transform` method.

There are two practicalities yet to define at this point. First, even though we defined the coupling layer as transforming the input from index $d+1$ to $D$, in practice it's usual to compose flows that alternate the dimensions that are transformed, with those that are not. Here we use binary masks of the form `[1, 0, 1, 0, ...]`. For the following coupling layer, we use the negated mask `[0, 1, 0, 1, ...]`. This gives a chance to the flow for transforming all dimensions in the input.

The second practicality has to do with the base distribution $p_u(\mathbf{u})$. If we use a standard Gaussian distribution, the log-probability is
$$
\begin{align}
\log p_u(\mathbf{u}) &= \log\left(\frac{1}{(2\pi)^{D/2}}\exp\left\lbrace-\frac{1}{2}\mathbf{u}^\top\mathbf{u}\right\rbrace\right) \\
&= -\frac{D}{2}\log(2\pi) - \frac{1}{2}\mathbf{u}^\top\mathbf{u}
\end{align}
$$

We now have all the ingredients to implement the Normalizing Flow:

```python
import torch
import numpy as np

class NormalizingFlow(nn.Module):
    def __init__(self, dim, num_flows, num_hidden):
        super().__init__()

        # Create mask with alternating pattern
        mask = torch.tensor([1, 0], dtype=torch.bool).repeat(dim // 2)

        self.layers = nn.ModuleList()
        for i in range(num_flows // 2):
            self.layers.append(Coupling(dim, num_hidden, mask))
            self.layers.append(Coupling(dim, num_hidden, mask))

        # The initial value of the log-determinant is 0
        self.register_buffer('init_log_det', torch.zeros(1))
        self.dim = dim

    def transform(self, z, inverse=False):
        log_det = self.init_log_det
        if not inverse:
            for layer in self.layers:
                z, log_det = layer(z, log_det)
        else:
            for layer in reversed(self.layers):
                z, log_det = layer(z, log_det, inverse=True)

        return z, log_det

    def forward(self, x):
        u, log_det = self.transform(x)
        log_pu = -0.5 * self.dim * (np.log(2.0 * np.pi) + (u ** 2).sum(dim=1))
        # Equation 2
        log_px = log_pu + log_det

        # Return the negative for minimization
        return -log_px

    def sample(self, num_samples):
        # Sample from base (Gaussian) distribution
        # and transform with flow
        u = torch.randn((num_samples, self.dim))
        x, _ = self.transform(u, inverse=True)
        return x
```

## Training the model

For visualization purposes, we will train a model for a very simple case where the data lies in a 2-dimensional space. Followin our implementation, the base distribution is a standard Gaussian:

```python
from utils import base, target, plot_density

plot_density(base)
```

The target is a bimodal Gaussian distribution, with one mode at $(1.5, 1.5)$, and another at $(-1.5, -1.5)$:

```python
plot_density(target)
```

For training, we will collect 2,000 samples from the target distribution.

```python
from torch.utils.data import TensorDataset, DataLoader

dataset = TensorDataset(target.sample(2000))
loader = DataLoader(dataset, batch_size=64)
```

Since the dimension of the space where the data lies is just 2 and the target density is not too complicated, the model shouldn't be too big. Here we will train a model composed of 4 transformations, where the 2-layer MLPs used for the coupling layers have 8 hidden units.

```python
flow = NormalizingFlow(dim=2, num_flows=4, num_hidden=8)
```

We're now ready to train the model! We will use Adam to train for 100 epochs. For each mini-batch, the value returned by the flow model is the negative log-likelihood for each sample. The loss that we minimize is then the mean across samples in the mini-batch.

```python
import matplotlib.pyplot as plt
from torch.optim import Adam
from tqdm.notebook import tqdm

optimizer = Adam(flow.parameters())

num_epochs = 100
num_iters = num_epochs * len(loader)
progress = tqdm(total=num_iters)

losses = np.zeros(num_epochs)

for epoch in range(1, num_epochs + 1):
    for data in loader:
        loss = flow(data[0]).mean()

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        desc = f'Epoch {epoch}/{num_epochs} - loss: {loss.item():.3f}'
        progress.set_description_str(desc)
        progress.update()

        losses[epoch - 1] += loss.item()

    losses[epoch - 1] /= len(loader)

progress.close()
plt.plot(losses)
plt.xlabel('Epochs')
plt.ylabel('Negative log-likelihood');
```

    HBox(children=(FloatProgress(value=0.0, max=3200.0), HTML(value='')))

Now that the model is trained, we can draw samples from it. Since these are samples in 2-D space, we can draw them in a scatter plot, with some crosses to indicate the locations of the two modes in the target density:

```python
samples = flow.sample(200).detach().cpu()
plt.scatter(samples[:, 0], samples[:, 1], label='Samples')
plt.scatter([1.5, -1.5], [1.5, -1.5], label='True modes')
plt.legend();
```

It seems the model has learned about the two modes!

As mentioned before, an interesting property of this use of normalizing flows is that in addition of generating samples, we can also evaluate the density at any point in space. We can use this to evaluate the density on a fine grid of points to have a better understanding of the density learned by the model.

```python
from utils import make_mesh, plot_surface

x1, x2, x = make_mesh()

# The model returns the negative log-density,
# so we negate it and take the exponential
log_px = -flow(x.reshape(-1, 2))
log_px = log_px.reshape(x.shape[:2]).detach()
px = log_px.exp()

plot_surface(x1, x2, px)
```

Alternatively, we can plot a contour map, as if we were looking at the density from above:

```python
plt.imshow(px)
plt.axis('off');
```

In spite of some artifacts in the center of the region, the model has clearly learned a density with two main modes located at the correct positions.

## Wrapping it up

Normalizing flows are a general mechanism that allows us to model complicated distributions, when we have access to a simple one. They have been applied to problems of variational inference, where they can serve as flexible approximate posteriors [1, 2, 3], and also for density estimation, particularly applied to image data [4, 5]. Among other topics, current work is focusing on building more expressive, yet still tractable normalizing flows [6, 7], and other formulations, such as continuous flows that are modeled as an ordinary differential equation [8]. If you are interested in diving deeper into this area, I highly recommend two recent surveys on the matter [9, 10].

### References

[1] Rezende, D. J., & Mohamed, S. (2015). Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770.

[2] Berg, R. V. D., Hasenclever, L., Tomczak, J. M., & Welling, M. (2018). Sylvester normalizing flows for variational inference. arXiv preprint arXiv:1803.05649.

[3] Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., & Welling, M. Improving variational inference with inverse autoregressive flow. (NeurIPS), 2016. URL http://arxiv.org/abs/1606.04934.

[4] Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2016). Density estimation using real nvp. arXiv preprint arXiv:1605.08803.

[5] Kingma, D. P., & Dhariwal, P. (2018). Glow: Generative flow with invertible 1x1 convolutions. In Advances in neural information processing systems (pp. 10215-10224).

[6] Huang, C. W., Krueger, D., Lacoste, A., & Courville, A. (2018). Neural autoregressive flows. arXiv preprint arXiv:1804.00779.

[7] De Cao, N., Titov, I., & Aziz, W. (2019). Block neural autoregressive flow. arXiv preprint arXiv:1904.04676.

[8] Grathwohl, W., Chen, R. T., Bettencourt, J., Sutskever, I., & Duvenaud, D. (2018). FFJORD: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367.

[9] Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., & Lakshminarayanan, B. (2019). Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02762.

[10] Kobyzev, I., Prince, S., & Brubaker, M. (2020). Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence.


# Lecture 11: Numerical optimization

[Download on GitHub](https://github.com/NumEconCopenhagen/lectures-2020)

[](https://mybinder.org/v2/gh/NumEconCopenhagen/lectures-2020/master?urlpath=lab/tree/11/Numerical_optimization.ipynb)

1. [Introduction](#Introduction)
2. [Example: The rosenbrock function](#Example:-The-rosenbrock-function)
3. [Constraints](#Constraints)
4. [Interpolation](#Interpolation)
5. [Dynamic optimization problems](#Dynamic-optimization-problems)
6. [Summary](#Summary)

You will learn to solve non-convex multi-dimensional optimization problems using numerical optimization with multistart and nesting (**scipy.optimize**). You will learn simple function approximation using linear interpolation (**scipy.interp**).

**Links:**

1. **scipy.optimize:** [overview](https://docs.scipy.org/doc/scipy/reference/optimize.html) + [tutorial](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html)
2. **scipy.interp:** [overview](https://docs.scipy.org/doc/scipy/reference/interpolate.html) + [tutorial](https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html)

**Useful note:** [Numerical Optimization in MATLAB](http://web.econ.ku.dk/munk-nielsen/notes/noteOptimization.pdf) (by Anders Munk-Nielsen)

```python
import numpy as np
import scipy as sp
from scipy import linalg
from scipy import optimize
from scipy import interpolate
import sympy as sm

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
```

# 1. Introduction

All **optimization problems** are characterized by:

1. Control vector (choices), $\boldsymbol{x} \in \mathbb{R}^k$
2. Objective function (payoff) to minimize, $f:\mathbb{R}^k \rightarrow \mathbb{R}$ (differentiable or not)
3. Constraints, i.e. $\boldsymbol{x}  \in C \subseteq \mathbb{R}^k$ (linear or non-linear interdependence)

**Maximization** is just **minimization** of $-f$.

> Note that $f$ might also take other inputs (parameters or a dataset), but these are fixed, and therefore not variables we optimize over.

All **optimizers** (minimizers) follow the structure:

1. Make initial guess
2. Evaluate the function (and perhaps gradients)
3. Check for convergence
4. Update guess and return to step 2

**Convergence:** "Small" change in function value since last iteration (or zero gradient).

**Characteristics** of optimizers:

1. Use gradients or not.
2. Allow for specifying bounds.
3. Allow for specifying general constraints.

**Gradients** provide useful information, but can be costly to compute (using analytical formula or numerically).

**Penalty terms** can (sometimes) instead be used to enforce bounds and constraints.

**Optimizers** you should know:

1. **Nelder-Mead:**
 * **Pro:** Robust (to e.g. noise in objective function) and does not require derivatives.
 * **Con:** Slow convergence. No bounds or constraints.
* **Newton-CG:**
 * **Pro:** Require few iterations. Very precise with analytical hessian for smooth functions.
 * **Con:** Costly computation of hessian. No bounds or constraints.
* **BFGS:** (like newton, but with smart computation of hessian)
  * **Pro:** Require few function evaluations.
  * **Con:** No bounds or constraints.
* **L-BFGS-B:** Like BFGS, but allows for bounds.
* **SLSQP:**
  * **Pro:** Bounds and constraints in multiple dimensions.
  * **Con:** Not as efficient as BFGS.

## 1.1 Gradient based optimizers

Let us look at the idea behind gradient based optimizers.

**One dimensional intuition:** Consider the second-order Taylor approximation around $x_n$:

$$
f_T(x) = f_T(x_n + \Delta x) \approx f(x_n)+ f^{\prime}(x_n) \Delta x + \frac{1}{2} f^{\prime\prime}(x_n) (\Delta x)^2
$$

Find the minimum wrt. to $\Delta x$ by solving the FOC:

$$
0 = \frac{d}{d\Delta x} f_T(x) = f^{\prime}(x_n) + f^{\prime\prime}(x_n) \Delta x \Leftrightarrow \Delta x = -\frac{f^{\prime}(x_n)}{f^{\prime\prime}(x_n)}
$$

**Algorithm:** `minimize_newton()`

1. Choose tolerance $\epsilon>0$, guess on $\boldsymbol{x}_0$, compute $f(\boldsymbol{x}_0)$, and set $n=1$.
2. Compute $\nabla f(\boldsymbol{x}_{n-1})$ (gradient/jacobian) and $\boldsymbol{H}f(\boldsymbol{x}_{n-1})$ (hessian).
3. Compute new guess

  $$
  \boldsymbol{x}_{n} = \boldsymbol{x}_{n-1} - [\boldsymbol{H}f(\boldsymbol{x}_{n-1})]^{-1} \nabla f(\boldsymbol{x}_{n-1})
  $$

3. If $|f(\boldsymbol{x}_n)-f(\boldsymbol{x}_{n-1})| < \epsilon$ then stop.
5. Set $n = n + 1$ and return to step 2.

```python
def minimize_newton(f,x0,jac,hess,max_iter=500,tol=1e-8):
    """ minimize function with Newtons' algorithm

    Args:

        f (callable): function
        x0 (float): initial value
        jac (callable): jacobian
        hess (callable): hessian
        max_iter (int): maximum number of iterations
        tol (float): tolerance

    Returns:

        x (float): root
        n (int): number of iterations used

    """

    # step 1: initialize
    x = x0
    fx = f(x0)
    n = 1

    # step 2-5: iteration
    while n < max_iter:

        x_prev = x
        fx_prev = fx

        # step 2: evaluate gradient and hessian
        jacx = jac(x_prev)
        hessx = hess(x_prev)

        # step 3: update x
        inv_hessx = linalg.inv(hessx)
        x = x_prev - inv_hessx@jacx

        # step 4: check convergence
        fx = f(x)
        if abs(fx-fx_prev) < tol:
            break

        # step 5: increment n
        n += 1

    return x,n
```

**Algorithm:** `minimize_gradient_descent()`

1. Choose tolerance $\epsilon>0$, potential step sizes, $ \boldsymbol{\alpha} = [\alpha_0,\alpha_1,\dots,\alpha_\#]$, guess on $\boldsymbol{x}_0$, compute $f(\boldsymbol{x}_0)$, and set $n=1$.
2. Compute $\nabla f(\boldsymbol{x}_{n-1})$.
3. Find good step size:

  $$
  \alpha^{\ast} = \arg \min_{\alpha \in \boldsymbol{\alpha}}  f(\boldsymbol{x}_{n-1} - \alpha \nabla f(\boldsymbol{x}_{n-1})
  $$

4. Compute new guess:

  $$
  \boldsymbol{x}_{n} = \boldsymbol{x}_{n-1} - \alpha^{\ast} \nabla f(\boldsymbol{x}_{n-1})
  $$

5. If $|f(\boldsymbol{x}_n)-f(\boldsymbol{x}_{n-1})| < \epsilon$ then stop.
6. Set $n = n + 1$ and return to step 2.

```python
def minimize_gradient_descent(f,x0,jac,alphas=[0.01,0.05,0.1,0.25,0.5,1],max_iter=500,tol=1e-8):
    """ minimize function with gradient descent

    Args:

        f (callable): function
        x0 (float): initial value
        jac (callable): jacobian
        alpha (list): potential step sizes
        max_iter (int): maximum number of iterations
        tol (float): tolerance

    Returns:

        x (float): root
        n (int): number of iterations used

    """

    # step 1: initialize
    x = x0
    fx = f(x0)
    n = 1

    # step 2-6: iteration
    while n < max_iter:

        x_prev = x
        fx_prev = fx

        # step 2: evaluate gradient
        jacx = jac(x)

        # step 3: find good step size (line search)
        fx_ast = np.inf
        alpha_ast = np.nan
        for alpha in alphas:
            x = x_prev - alpha*jacx
            fx = f(x)
            if fx < fx_ast:
                fx_ast = fx
                alpha_ast = alpha

        # step 4: update guess
        x = x_prev - alpha_ast*jacx

        # step 5: check convergence
        fx = f(x)
        if abs(fx-fx_prev) < tol:
            break

        # d. update i
        n += 1

    return x,n
```

**Many generalizations:**

1. Use both Hessian and line search
2. Stop line search when improvement found
3. Limit attention to a "trust-region"

etc. etc. etc. etc.

# 2. Example: The rosenbrock function

Consider the **rosenbrock function**:

$$
f(\boldsymbol{x}) = f(x_1,x_2) =0.5(1-x_{1})^{2}+(x_{2}-x_{1}^{2})^{2}
$$

with **jacobian** (gradient)

$$
\nabla f(\boldsymbol{x})=\begin{bmatrix}\frac{\partial f}{\partial x_{1}}\\
\frac{\partial f}{\partial x_{2}}
\end{bmatrix}=\begin{bmatrix}-(1-x_{1})-4x_{1}(x_{2}-x_{1}^{2})\\
2(x_{2}-x_{1}^{2})
\end{bmatrix}
$$

and **hessian**:

$$
\boldsymbol{H}f(\boldsymbol{x})=\begin{bmatrix}\frac{\partial f}{\partial x_{1}x_{1}} & \frac{\partial f}{\partial x_{1}x_{2}}\\
\frac{\partial f}{\partial x_{1}x_{2}} & \frac{\partial f}{\partial x_{2}x_{2}}
\end{bmatrix}=\begin{bmatrix}1-4x_{2}+12x_{1}^{2} & -4x_{1}\\
-4x_{1} & 2
\end{bmatrix}
$$

**Note:** Minimum is at $(1,1)$ where $f(1,1)=0$.

**Check jacobian and hessian:**

```python
sm.init_printing(use_unicode=True)
x1 = sm.symbols('x_1')
x2 = sm.symbols('x_2')
f = 0.5*(1.0-x1)**2 + (x2-x1**2)**2
```

```python
Df = sm.Matrix([sm.diff(f,i) for i in [x1,x2]])
Df
```

```python
Hf = sm.Matrix([[sm.diff(f,i,j) for j in [x1,x2]] for i in [x1,x2]])
Hf
```

**Implementation:**

```python
def _rosen(x1,x2):
    return 0.5*(1.0-x1)**2+(x2-x1**2)**2
def rosen(x):
    return _rosen(x[0],x[1])
def rosen_jac(x):
    return np.array([-(1.0-x[0])-4*x[0]*(x[1]-x[0]**2),2*(x[1]-x[0]**2)])
def rosen_hess(x):
    return np.array([[1-4*x[1]+12*x[0]**2,-4*x[0]],[-4*x[0],2]])
```

**3D Plot:**

```python
# a. grids
x1_vec = np.linspace(-2,2,500)
x2_vec = np.linspace(-2,2,500)
x1_grid,x2_grid = np.meshgrid(x1_vec,x2_vec,indexing='ij')
rosen_grid = _rosen(x1_grid,x2_grid)

# b. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(x1_grid,x2_grid,rosen_grid,cmap=cm.jet)

# c. add labels
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$u$')

# d. invert xaxis
ax.invert_xaxis()

# e. add colorbar
fig.colorbar(cs);
```

**Contour plot:**

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
levels = [1e-6,5*1e-6,1e-5,5*1e-5,1e-4,5*1e-4,1e-3,5*1e-3,1e-2,5*1e-2,1,2,4,6,8,12,16,20]
cs = ax.contour(x1_grid,x2_grid,rosen_grid,levels=levels,cmap=cm.jet)
fig.colorbar(cs);
```

**Newton:**

```python
x0 = np.array([5,4])
x,n = minimize_newton(rosen,x0,rosen_jac,rosen_hess)
print(n,x,rosen(x))
```

    6 [1. 1.] 0.0

**Gradient descent:**

```python
x0 = np.array([5,4])
x,n = minimize_gradient_descent(rosen,x0,rosen_jac,alphas=[0.01,0.05,0.1,0.25,0.5,1])
print(n,x,rosen(x))
```

    173 [1.00020519 1.00053964] 3.7750814497569406e-08

## 2.1 Scipy minimizers

**Preperation I:** Function for collecting infomation while running optimizing:

```python
# complicated -> not necessary to understand it
def collect(x):

    # globals used to keep track across iterations
    global evals # set evals = 0 before calling optimizer
    global x0
    global x1s
    global x2s
    global fs

    # a. initialize list
    if evals == 0:
        x1s = [x0[0]]
        x2s = [x0[1]]
        fs = [rosen(x0)]

    # b. append trial values
    x1s.append(x[0])
    x2s.append(x[1])
    fs.append(rosen(x))

    # c. increment number of evaluations
    evals += 1
```

**Preperation II:** Function plotting the collected information:

```python
# complicated -> not necessary to understand it
def contour():

    global evals
    global x1s
    global x2s
    global fs

    # a. contour plot
    fig = plt.figure(figsize=(10,4))
    ax = fig.add_subplot(1,2,1)
    levels = [1e-6,5*1e-6,1e-5,5*1e-5,1e-4,5*1e-4,1e-3,5*1e-3,1e-2,5*1e-2,1,2,4,6,8,12,16,20]
    cs = ax.contour(x1_grid,x2_grid,rosen_grid,levels=levels,cmap=cm.jet)
    fig.colorbar(cs)
    ax.plot(x1s,x2s,'-o',ms=4,color='black')
    ax.set_xlabel('$x_1$')
    ax.set_ylabel('$x_2$')

    # b. function value
    ax = fig.add_subplot(1,2,2)
    ax.plot(np.arange(evals+1),fs,'-o',ms=4,color='black')
    ax.set_xlabel('iteration')
    ax.set_ylabel('function value')
```

**Nelder-Mead**

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,
                           method='Nelder-Mead',
                           callback=collect, # call the collect() before each iteration
                           options={'disp':True}) # display the results
contour()
```

> **Note:** Does not require a gradient. Slow convergence close to target.

**Newton** (with analytical hessian)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,hess=rosen_hess,
                           method='Newton-CG',
                           callback=collect,
                           options={'disp':True})
contour()
```

> **Note:** Smoother and faster.

**Newton** (with numerical hessian computed by scipy)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,
                           method='Newton-CG',
                           callback=collect,
                           options={'disp':True})
contour()
```

> **Note:** Same as above, but gradient evaluations instead of hessian evaluations.

**BFGS** (with analytical gradient)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,
                           method='BFGS',
                           callback=collect,
                           options={'disp':True})
contour()
```

> **Note:** Non-smooth, but fast. Very low number of function evaluations.

**BFGS** (with numerical gradient computed by scipy)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,
                           method='BFGS',
                           callback=collect,
                           options={'disp':True})
contour()
```

> **Note:** Same as above, but more function evaluations.

**L-BFGS-B** (with analytical gradient)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,
                           method='L-BFGS-B',
                           bounds=((-3,3),(-3,3)),
                           callback=collect,
                           options={'disp':True})
contour()
```

**SLSQP**

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,
                           method='SLSQP',
                           bounds=((-2,2),(-2,2)),
                           callback=collect,
                           options={'disp':True})
contour()
```

## 2.2 Controling the optimizers

> **Note:** See the settings for each optimizer in the [documention](https://docs.scipy.org/doc/scipy/reference/optimize.html).

We can lower the **tolerance**:

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,
                           method='BFGS',
                           callback=collect,
                           options={'disp':True,'gtol':1e-2}) # note this
# contour()
```

    Optimization terminated successfully.
             Current function value: 0.000006
             Iterations: 11
             Function evaluations: 48
             Gradient evaluations: 12

We can change the **maximum number of iterations**:

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,
                           method='BFGS',
                           callback=collect,
                           options={'disp':True,'maxiter':5}) # note this and warning
contour()
```

## 2.3 Sombrero function: Local minima and multistart

Consider the **sombrero** function

$$
f(x_1,x_2) = g\Big(\sqrt{x_1^2 + x_2^2}\Big)
$$

where

$$
g(r) = -\frac{\sin(r)}{r+10^{-4}} + 10^{-4}r^2
$$

The **global minimum** of this function is (0,0). But the function also have (infinitely many) **local minima**. How to avoid these?

```python
def _sombrero(x1,x2):
    r = np.sqrt(x1**2 + x2**2)
    return -np.sin(r)/(r+1e-4) + 1e-4*r**2

sombrero = lambda x: _sombrero(x[0],x[1])
```

## 2.4 3D plot

```python
# a. grids
x1_vec = np.linspace(-15,15,500)
x2_vec = np.linspace(-15,15,500)
x1_grid_sombrero,x2_grid_sombrero = np.meshgrid(x1_vec,x2_vec,indexing='ij')
sombrero_grid = _sombrero(x1_grid_sombrero,x2_grid_sombrero)

# b. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(x1_grid_sombrero,x2_grid_sombrero,sombrero_grid,cmap=cm.jet)

# c. add labels
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$u$')

# d. invert xaxis
ax.invert_xaxis()

# e. colorbar
fig.colorbar(cs);
```

## 2.5 Multistart - BFGS

**Multi-start:** Draw many random starting values:

```python
np.random.seed(1986)
x0s = -15 + 30*np.random.uniform(size=(5000,2)) # in [-15,15]
xs = np.empty((5000,2))
fs = np.empty(5000)
```

Try to solve with **BFGS** starting from each of these:

```python
fopt = np.inf
xopt = np.nan
for i,x0 in enumerate(x0s):

    # a. optimize
    result = optimize.minimize(sombrero,x0,method='BFGS')
    xs[i,:] = result.x
    f = result.fun

    # b. print first 10 or if better than seen yet
    if i < 10 or f < fopt: # plot 10 first or if improving
        if f < fopt:
            fopt = f
            xopt = xs[i,:]

        print(f'{i:4d}: x0 = ({x0[0]:6.2f},{x0[0]:6.2f})',end='')
        print(f' -> converged at ({xs[i][0]:6.2f},{xs[i][0]:6.2f}) with f = {f:.12f}')

# best solution
print(f'\nbest solution:\n x = ({xopt[0]:6.2f},{xopt[1]:6.2f}) -> f = {fopt:.12f}')
```

       0: x0 = (  2.07,  2.07) -> converged at (  2.26,  2.26) with f = -0.051182722123
       1: x0 = (  3.25,  3.25) -> converged at (  3.69,  3.69) with f = -0.051182722111
       2: x0 = (  1.35,  1.35) -> converged at (  1.67,  1.67) with f = -0.122414211497
       3: x0 = ( -3.42, -3.42) -> converged at ( -4.63, -4.63) with f = -0.122414211494
       4: x0 = (  5.70,  5.70) -> converged at (  5.06,  5.06) with f = -0.122414211497
       5: x0 = (  5.71,  5.71) -> converged at (  4.06,  4.06) with f = -0.122414211468
       6: x0 = ( -6.49, -6.49) -> converged at ( -4.96, -4.96) with f = -0.122414211497
       7: x0 = ( -5.77, -5.77) -> converged at ( -7.34, -7.34) with f = -0.122414211356
       8: x0 = (  1.58,  1.58) -> converged at (  0.04,  0.04) with f = -0.997762360170
       9: x0 = (  2.33,  2.33) -> converged at (  2.72,  2.72) with f = -0.051182722123
      25: x0 = (  1.41,  1.41) -> converged at (  0.03,  0.03) with f = -0.997762360171
     150: x0 = (-14.53,-14.53) -> converged at ( -0.06, -0.06) with f = -0.997762360171
     223: x0 = (  2.87,  2.87) -> converged at ( -0.07, -0.07) with f = -0.997762360171
     279: x0 = ( -6.90, -6.90) -> converged at ( -0.04, -0.04) with f = -0.997762360171
     354: x0 = ( -1.24, -1.24) -> converged at ( -0.03, -0.03) with f = -0.997762360171
    1588: x0 = (-12.93,-12.93) -> converged at ( -0.05, -0.05) with f = -0.997762360171
    4700: x0 = (  1.06,  1.06) -> converged at (  0.06,  0.06) with f = -0.997762360171

    best solution:
     x = (  0.06,  0.04) -> f = -0.997762360171

The solver, wrongly, **converges to many of the local minima**:

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(xs[:,0],xs[:,1]);
```

## 2.6 Multistart - Nelder-Mead

Try to solve with **Nelder-Mead** starting from each of these:

```python
fopt = np.inf
xopt = np.nan
for i,x0 in enumerate(x0s):

    # a. optimize
    result = optimize.minimize(sombrero,x0,method='Nelder-Mead')
    xs[i,:] = result.x
    f = result.fun

    # b. print first 10 or if better than seen yet
    if i < 10 or f < fopt: # plot 10 first or if improving
        if f < fopt:
            fopt = f
            xopt = xs[i,:]
        print(f'{i:4d}: x0 = ({x0[0]:6.2f},{x0[0]:6.2f})',end='')
        print(f' -> converged at ({xs[i][0]:6.2f},{xs[i][1]:6.2f}) with f = {f:.12f}')

# best solution
print(f'\nbest solution:\n x = ({xopt[0]:6.2f},{xopt[1]:6.2f}) -> f = {fopt:.12f}')
```

       0: x0 = (  2.07,  2.07) -> converged at (  2.20,-13.85) with f = -0.051182722123
       1: x0 = (  3.25,  3.25) -> converged at (  3.43, 13.60) with f = -0.051182722123
       2: x0 = (  1.35,  1.35) -> converged at (  1.39,  7.59) with f = -0.122414211497
       3: x0 = ( -3.42, -3.42) -> converged at ( -3.71,  6.76) with f = -0.122414211497
       4: x0 = (  5.70,  5.70) -> converged at (  6.69,  3.84) with f = -0.122414211497
       5: x0 = (  5.71,  5.71) -> converged at (  4.98, 13.11) with f = -0.051182722123
       6: x0 = ( -6.49, -6.49) -> converged at ( -7.34,  2.37) with f = -0.122414211497
       7: x0 = ( -5.77, -5.77) -> converged at ( -7.45, -2.01) with f = -0.122414211497
       8: x0 = (  1.58,  1.58) -> converged at ( -0.02,  0.06) with f = -0.997762360171
       9: x0 = (  2.33,  2.33) -> converged at (  2.36, 13.83) with f = -0.051182722123

    best solution:
     x = ( -0.02,  0.06) -> f = -0.997762360171

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(xs[:,0],xs[:,1]);
```

# 3. Constraints

Consider the **constrained problem**:

$$
\min_{x_1,x_2,x_3,x_4} x_1x_4(x_1+x_2+x_3) + x_3
$$

subject to

$$
\begin{aligned}
x_1x_2x_3x_4 &\geq 25 \\
x_1^2x_2^2x_3^2x_4^2 &= 40 \\
1 \leq x_1,x_2,x_3,x_4 &\leq 5
\end{aligned}
$$

Define **objective** and **constraints**:

```python
def _objective(x1,x2,x3,x4):
    return x1*x4*(x1+x2+x3)+x3

def objective(x):
    return _objective(x[0],x[1],x[2],x[3])

def ineq_constraint(x):
    return x[0]*x[1]*x[2]*x[3]-25.0

def eq_constraint(x):
    sum_eq = 40.0
    for i in range(4):
        sum_eq = sum_eq - x[i]**2
    return sum_eq
```

```python
# a. setup
bound = (1.0,5.0)
bounds = (bound, bound, bound, bound)
ineq_con = {'type': 'ineq', 'fun': ineq_constraint}
eq_con = {'type': 'eq', 'fun': eq_constraint}

# b. call optimizer
x0 = (40**(1/8),40**(1/8),40**(1/8),40**(1/8)) # fit the equality constraint
result = optimize.minimize(objective,x0,
                             method='SLSQP',
                             bounds=bounds,
                             constraints=[ineq_con,eq_con],
                             options={'disp':True})

print('\nx = ',result.x)
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 17.01401728904412
                Iterations: 9
                Function evaluations: 55
                Gradient evaluations: 9

    x =  [1.         4.74299967 3.82114994 1.3794083 ]

**Alternative:** Extend the **objective function with a penalty term**, where guesses outside the allowed bounds and constraints are projected into the allowed region, but a (large) penalty term is added to discourage this. Solve this problem with an unconstrained solver. Not always easy to do in practice if there are multiple constraint (see lecture 3 for a simple example).

# 4. Interpolation

**Intermezzo:** To consider dynamic optimization problems, we need to think about interpolation.

**Inputs:**

1. Sorted vector of known points (grid vector), $G$
2. Vector of known values (at these points), $F$
3. A new point, `x`

**Algorithm:** `linear_interpolate()`
1. Determine `i`  such that

$$
G_i \leq x < G_{i+1}
$$

2. Compute interpolated value by

$$
y =  F_{i} + \frac{F_{i+1}-F_{i}}{G_{i+1}-G_{i}}(x-G_{i})
$$

**Extrapolation:**

1. Below where $x < G_0$:

$$
y =  F_{0} + \frac{F_{1}-F_{0}}{G_{1}-G_{0}}(x-G_{0})
$$

2. Below where $x > G_n$:

$$
y =  F_{n-1} + \frac{F_{n}-F_{n-1}}{G_{n}-G_{n-1}}(x-G_{n-1})
$$

```python
def linear_interpolate(G,F,x):
    """ linear interpolation (and extrapolation)

    Args:

        G (ndarray): known points
        F (ndarray): known values
        x (float): point to be interpolated

    Returns:

        y (float): intepolated value

    """

    assert len(G) == len(F)
    n = len(G)

    # a. find index in known points
    if x < G[1]: # exprapolation below
        i = 0
    elif x > G[-2]: # extrapolation above
        i = n-2
    else: # true interpolation

        # search
        i = 0
        while x >= G[i+1] and i < n-1:
            i += 1

        assert x >= G[i]
        assert x < G[i+1]

    # b. interpolate
    diff_G = G[i+1]-G[i]
    diff_F = F[i+1]-F[i]
    slope = diff_F/diff_G
    y = F[i] + slope*(x-G[i])

    return y
```

## 4.1 Example

Consider the following function and known points:

```python
f = lambda x: (x-3)**3 - 3*x**2 + 5*x

G = np.linspace(-5,10,6)
F = f(G)
```

**Simple test:**

```python
for x in [-2.3,4.1,7.5,9.1]:
    true = f(x)
    y = linear_interpolate(G,F,x)
    print(f'x = {x:4.1f} -> true = {true:6.1f}, interpolated = {y:6.1f}')
```

    x = -2.3 -> true = -176.2, interpolated = -193.5
    x =  4.1 -> true =  -28.6, interpolated =  -27.7
    x =  7.5 -> true =  -40.1, interpolated =  -24.5
    x =  9.1 -> true =   24.1, interpolated =   50.7

**Scipy.interpolate:** Use the *RegularGridInterpolator*

```python
# a. construct interpolation function
interp_func = interpolate.RegularGridInterpolator([G],F,
                                                  bounds_error=False,
                                                  fill_value=None)

# bounds_error=False and fill_value=None allow for extrapolation

# b. interpolate
grid = np.linspace(-7,12,500)
interp_values = interp_func(grid)

# c. evaluate true values
true_values = f(grid)

# d. plot true and interpolated values
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(G,F,'o',label='known points')
ax.plot(grid,true_values,'-',lw=1,label='true function')
ax.plot(grid,interp_values,'-',lw=1,label='interpolated values')
ax.legend(loc='lower right',facecolor='white',frameon=True);
```

**Note:**

1. Linear interpolation works best when the function does not curve too much.
2. Extrapolation is much worse than interpolation.

**Multiple dimensions:** Same principle, ``interpolate.RegularGridInterpolator([G1,G2,G3],F)``.

# 5. Dynamic optimization problems

The following subject is hard. But also extremely useful. *If you master this, you can solve (almost) all economic models you meet on your way in life*.

## 5.1 Problem formulation

Consider a **household** living in two periods.

In the **second period** it gets utility from **consuming** and **leaving a bequest** (warm glow),

$$
\begin{aligned}
v_{2}(m_{2})&= \max_{c_{2}}\frac{c_{2}^{1-\rho}}{1-\rho}+\nu\frac{(m_{2}-c_{2}+\kappa)^{1-\rho}}{1-\rho}\\
\text{s.t.} \\
c_{2} &\in [0,m_{2}]
\end{aligned}
$$

where

* $m_2$ is cash-on-hand
* $c_2$ is consumption
* $\rho > 1$ is the risk aversion coefficient
* $\nu > 0 $ is the strength of the bequest motive
* $\kappa > 0$ is the degree of luxuriousness in the bequest motive
* the constraint ensures the household *cannot* die in debt

The **value function** $v(m_2)$ measures the household's value of having $m_2$ at the beginning of period 2.

```python
def utility(c,rho):
    return c**(1-rho)/(1-rho)

def bequest(m,c,nu,kappa,rho):
    return nu*(m-c+kappa)**(1-rho)/(1-rho)

def v2(c2,m2,rho,nu,kappa):
    return utility(c2,rho) + bequest(m2,c2,nu,kappa,rho)
```

In the **first period**, the household gets utility from consuming and takes into account that it will also live in the next-period, where it receives a stochastic income,

$$
\begin{aligned}
v_{1}(m_{1})&=\max_{c_{1}}\frac{c_{1}^{1-\rho}}{1-\rho}+\beta\mathbb{E}_{1}\left[v_2(m_2)\right]\\&\text{s.t.}&\\
m_2&= (1+r)(m_{1}-c_{1})+y_{2} \\
y_{2}&= \begin{cases}
1-\Delta & \text{with prob. }0.5\\
1+\Delta & \text{with prob. }0.5
\end{cases}\\
c_{1}&\in [0,m_{1}]\\
\end{aligned}
$$

where

* $m_1$ is cash-on-hand in period 1
* $c_1$ is consumption in period 1
* $\beta > 0$ is the discount factor
* $\mathbb{E}_1$ is the expectation operator conditional on information in period 1
* $y_2$ is income in period 2
* $\Delta \in (0,1)$ is the level of income risk (mean-preserving)
* $r$ is the return on savings
* the last constraint ensures the household *cannot* borrow

```python
def v1(c1,m1,rho,beta,r,Delta,v2_interp):

    # a. v2 value, if low income
    m2_low = (1+r)*(m1-c1) + 1-Delta
    v2_low = v2_interp([m2_low])[0]

    # b. v2 value, if high income
    m2_high = (1+r)*(m1-c1) + 1+Delta
    v2_high = v2_interp([m2_high])[0]

    # c. expected v2 value
    v2 = 0.5*v2_low + 0.5*v2_high

    # d. total value
    return utility(c1,rho) + beta*v2
```

## 5.2 Solve household problem

Choose **parameters**:

```python
rho = 8
kappa = 0.5
nu = 0.1
r = 0.04
beta = 0.94
Delta = 0.5
```

**Solve second period:**

```python
def solve_period_2(rho,nu,kappa,Delta):

    # a. grids
    m2_vec = np.linspace(1e-8,5,500)
    v2_vec = np.empty(500)
    c2_vec = np.empty(500)

    # b. solve for each m2 in grid
    for i,m2 in enumerate(m2_vec):

        # i. objective
        obj = lambda c2: -v2(c2,m2,rho,nu,kappa)

        # ii. initial value (consume half)
        x0 = m2/2

        # iii. optimizer
        result = optimize.minimize_scalar(obj,x0,method='bounded',bounds=[1e-8,m2])

        # iv. save
        v2_vec[i] = -result.fun
        c2_vec[i] = result.x

    return m2_vec,v2_vec,c2_vec

# solve
m2_vec,v2_vec,c2_vec = solve_period_2(rho,nu,kappa,Delta)

# illustration
fig = plt.figure(figsize=(10,4))
ax = fig.add_subplot(1,2,1)
ax.plot(m2_vec,c2_vec)
ax.set_xlabel('$m_2$')
ax.set_ylabel('$c_2$')
ax.set_title('consumption function in period 2')

ax = fig.add_subplot(1,2,2)
ax.plot(m2_vec,v2_vec)
ax.set_xlabel('$m_2$')
ax.set_ylabel('$v_2$')
ax.set_title('value function in period 2')
ax.set_ylim([-40,1]);
```

**Question:** Why is there a kink in the consumption function?

**Construct interpolator:**

```python
v2_interp = interpolate.RegularGridInterpolator([m2_vec], v2_vec,
                                                bounds_error=False,fill_value=None)
```

**Solve first period:**

```python
def solve_period_1(rho,beta,r,Delta,v2_interp):

    # a. grids
    m1_vec = np.linspace(1e-8,4,100)
    v1_vec = np.empty(100)
    c1_vec = np.empty(100)

    # b. solve for each m1 in grid
    for i,m1 in enumerate(m1_vec):

        # i. objective
        obj = lambda c1: -v1(c1,m1,rho,beta,r,Delta,v2_interp)

        # ii. initial guess (consume half)
        x0 = m1*1/2

        # iii. optimize
        result = optimize.minimize_scalar(obj,x0,method='bounded',bounds=[1e-8,m1])

        # iv. save
        v1_vec[i] = -result.fun
        c1_vec[i] = result.x

    return m1_vec,v1_vec,c1_vec

# solve
m1_vec,v1_vec,c1_vec = solve_period_1(rho,beta,r,Delta,v2_interp)

# illustrate
fig = plt.figure(figsize=(10,4))
ax = fig.add_subplot(1,2,1)
ax.plot(m1_vec,c1_vec)
ax.set_xlabel('$m_1$')
ax.set_ylabel('$c_1$')
ax.set_title('consumption function in period 1')

ax = fig.add_subplot(1,2,2)
ax.plot(m1_vec,v1_vec)
ax.set_xlabel('$m_1$')
ax.set_ylabel('$c_1$')
ax.set_title('value function in period 1')
ax.set_ylim([-40,1]);
```

**Summary:** We can summarize what we have done in a single function doing:

1. Solve period 2 (i.e. find $v_2(m_2)$ og $c_2(m_2)$)
2. Construct interpolator of $v_2(m_2)$
3. Solve period 1 (i.e. find $v_1(m_1)$ og $c_1(m_1)$)

```python
def solve(rho,beta,r,Delta,nu,kappa):

    # a. solve period 2
    m2_vec,v2_vec,c2_vec = solve_period_2(rho,nu,kappa,Delta)

    # b. construct interpolator
    v2_interp = interpolate.RegularGridInterpolator([m2_vec], v2_vec,
        bounds_error=False,fill_value=None)

    # b. solve period 1
    m1_vec,v1_vec,c1_vec = solve_period_1(rho,beta,r,Delta,v2_interp)

    return m1_vec,c1_vec
```

**Plot consumption function for various level of income risk**, i.e varios $\Delta$

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

for Delta in [0.05,0.15,0.25]:
    m1_vec,c1_vec = solve(rho,beta,r,Delta,nu,kappa)
    ax.plot(m1_vec,c1_vec,label=f'$\Delta = {Delta}$')

ax.legend(loc='lower right',facecolor='white',frameon=True)
ax.set_xlabel('$m_1$')
ax.set_ylabel('$c_1$')
ax.set_title('value function in period 1')
ax.set_xlim([0,2])
ax.set_ylim([0,1.5]);
```

**Main takeaway:** The household lower its consumption when risk increases (such as in a recession). This is called **precautionary saving**.

# 6. Summary

**This lecture:**

1. Solving multidimensional optimization problems with and without gradients (and hessians)
2. Using multistart to alleviate problems with local minima (due to non-convexities)
3. Linear interpolation between known points
4. Solving dynamic optimization problems backwards period-by-period

**Dynamic optimization:** Extremely useful technique. Can handle multiple periods, multiple states and choices, more shocks etc. You can solve general equilibrium models where the households solve such problems.

**Next lecture:** The Need for Speed.


**Notebook Outline:**

- [Setup with libraries](#Set-up-Cells)
- [Fundamental equations for Poisson MGWR](#Fundamental-equations-for-Binomial-MGWR)
- [Example Dataset](#Example-Dataset)
- [Helper functions](#Helper-functions)
- [Univariate example](#Univariate-example)
    - [Parameter check](#Parameter-check)
    - [Bandwidths check](#Bandwidths-check)

### Set up Cells

```python
import sys
sys.path.append("C:/Users/msachde1/Downloads/Research/Development/mgwr")
```

```python
import warnings
warnings.filterwarnings("ignore")
import pandas as pd
import numpy as np

from mgwr.gwr import GWR
from spglm.family import Gaussian, Binomial, Poisson
from mgwr.gwr import MGWR
from mgwr.sel_bw import Sel_BW
import multiprocessing as mp
pool = mp.Pool()
from scipy import linalg
import numpy.linalg as la
from scipy import sparse as sp
from scipy.sparse import linalg as spla
from spreg.utils import spdot, spmultiply
from scipy import special
import libpysal as ps
import seaborn as sns
import matplotlib.pyplot as plt
from copy import deepcopy
import copy
from collections import namedtuple
import spglm
```

### Fundamental equations for Binomial MGWR

\begin{align}
l = log_b ( p / (1-p)) = ({\sum} {\beta} & _k x _{k,i}) \\
\end{align}

where  $x_{k,1}$ is the kth explanatory variable in place i, $𝛽_ks$ are the parameters and p is the probability such that p = P( Y = 1 ).

By exponentiating the log-odds:

$p / (1-p) = b^ {𝛽_0 + 𝛽_1 x_1 + 𝛽_2 x_2}$

It follows from this - the probability that Y = 1 is:

$p = (b^ {𝛽_0 + 𝛽_1 x_1 + 𝛽_2 x_2}) / (b^ {𝛽_0 + 𝛽_1 x_1 + 𝛽_2 x_2} + 1)$ = $1 / (1 + b^ {- 𝛽_0 + 𝛽_1 x_1 + 𝛽_2 x_2})$

### Example Dataset

#### Clearwater data - downloaded from link: https://sgsup.asu.edu/sparc/multiscale-gwr

```python
data_p = pd.read_csv("C:/Users/msachde1/Downloads/logistic_mgwr_data/landslides.csv")
coords = list(zip(data_p['X'],data_p['Y']))
y = np.array(data_p['Landslid']).reshape((-1,1))
elev = np.array(data_p['Elev']).reshape((-1,1))
slope = np.array(data_p['Slope']).reshape((-1,1))
SinAspct = np.array(data_p['SinAspct']).reshape(-1,1)
CosAspct = np.array(data_p['CosAspct']).reshape(-1,1)
X = np.hstack([elev,slope,SinAspct,CosAspct])
x = slope

X_std = (X-X.mean(axis=0))/X.std(axis=0)
x_std = (x-x.mean(axis=0))/x.std(axis=0)
y_std = (y-y.mean(axis=0))/y.std(axis=0)
```

### Helper functions
Hardcoded here for simplicity in the notebook workflow

Please note: A separate bw_func_b will not be required when changes will be made in the repository

```python
kernel='bisquare'
fixed=False
spherical=False
search_method='golden_section'
criterion='AICc'
interval=None
tol=1e-06
max_iter=200
X_glob=[]
```

```python
def gwr_func(y, X, bw,family=Gaussian(),offset=None):
    return GWR(coords, y, X, bw, family,offset,kernel=kernel,
               fixed=fixed, constant=False,
               spherical=spherical, hat_matrix=False).fit(
                   lite=True, pool=pool)

def bw_func_b(coords,y, X):
    selector = Sel_BW(coords,y, X,family=Binomial(),offset=None, X_glob=[],
                      kernel=kernel, fixed=fixed,
                      constant=False, spherical=spherical)
    return selector

def bw_func_p(coords,y, X):
    selector = Sel_BW(coords,y, X,family=Poisson(),offset=off, X_glob=[],
                      kernel=kernel, fixed=fixed,
                      constant=False, spherical=spherical)
    return selector

def bw_func(coords,y,X):
    selector = Sel_BW(coords,y,X,X_glob=[],
                      kernel=kernel, fixed=fixed,
                      constant=False, spherical=spherical)
    return selector

def sel_func(bw_func, bw_min=None, bw_max=None):
    return bw_func.search(
        search_method=search_method, criterion=criterion,
        bw_min=bw_min, bw_max=bw_max, interval=interval, tol=tol,
        max_iter=max_iter, pool=pool, verbose=False)
```

### Univariate example
#### GWR model with independent variable, x = slope

```python
bw_gwbr=Sel_BW(coords,y_std,x_std,family=Binomial(),constant=False).search()
```

```python
gwbr_model=GWR(coords,y_std,x_std,bw=bw_gwbr,family=Binomial(),constant=False).fit()
```

```python
bw_gwbr
```

    198.0

#### MGWR Binomial loop with one independent variable, x = slope

##### Edited multi_bw function - original function in https://github.com/pysal/mgwr/blob/master/mgwr/search.py#L167

```python
def multi_bw(init,coords,y, X, n, k, family=Gaussian(),offset=None, tol=1e-06, max_iter=200, multi_bw_min=[None], multi_bw_max=[None],rss_score=False,bws_same_times=3,
                     verbose=False):

    if multi_bw_min==[None]:
        multi_bw_min = multi_bw_min*X.shape[1]

    if multi_bw_max==[None]:
        multi_bw_max = multi_bw_max*X.shape[1]

    if isinstance(family,spglm.family.Poisson):
        bw = sel_func(bw_func_p(coords,y,X))
        optim_model=gwr_func(y,X,bw,family=Poisson(),offset=offset)
        err = optim_model.resid_response.reshape((-1, 1))
        param = optim_model.params
        #This change for the Poisson model follows from equation (1) above
        XB = offset*np.exp(np.multiply(param, X))

    elif isinstance(family,spglm.family.Binomial):
        bw = sel_func(bw_func_b(coords,y,X))
        optim_model=gwr_func(y,X,bw,family=Binomial())
        err = optim_model.resid_response.reshape((-1, 1))
        param = optim_model.params
        #This change for the Binomial model follows from equation above
        XB = 1/(1+np.exp(-1*np.multiply(optim_model.params,X)))
        #print(XB)

    else:
        bw=sel_func(bw_func(coords,y,X))
        optim_model=gwr_func(y,X,bw)
        err = optim_model.resid_response.reshape((-1, 1))
        param = optim_model.params
        XB = np.multiply(param, X)

    bw_gwr = bw
    XB=XB

    if rss_score:
        rss = np.sum((err)**2)
    iters = 0
    scores = []
    delta = 1e6
    BWs = []
    bw_stable_counter = np.ones(k)
    bws = np.empty(k)

    try:
        from tqdm.auto import tqdm  #if they have it, let users have a progress bar
    except ImportError:

        def tqdm(x, desc=''):  #otherwise, just passthrough the range
            return x

    for iters in tqdm(range(1, max_iter + 1), desc='Backfitting'):
        new_XB = np.zeros_like(X)
        params = np.zeros_like(X)

        for j in range(k):
            temp_y = XB[:, j].reshape((-1, 1))
            temp_y = temp_y + err
            temp_X = X[:, j].reshape((-1, 1))

            #The step below will not be necessary once the bw_func is changed in the repo to accept family and offset as attributes
            if isinstance(family,spglm.family.Poisson):

                bw_class = bw_func_p(coords,temp_y, temp_X)

            elif isinstance(family,spglm.family.Binomial):

                bw_class = bw_func_b(coords,temp_y, temp_X)

            else:
                bw_class = bw_func(coords,temp_y, temp_X)

            if np.all(bw_stable_counter == bws_same_times):
                #If in backfitting, all bws not changing in bws_same_times (default 3) iterations
                bw = bws[j]
            else:
                bw = sel_func(bw_class, multi_bw_min[j], multi_bw_max[j])
                if bw == bws[j]:
                    bw_stable_counter[j] += 1
                else:
                    bw_stable_counter = np.ones(k)

            #Changed gwr_func to accept family and offset as attributes
            optim_model = gwr_func(temp_y, temp_X, bw,family,offset)
            err = optim_model.resid_response.reshape((-1, 1))
            param = optim_model.params.reshape((-1, ))
            new_XB[:, j] = optim_model.predy.reshape(-1)
            params[:, j] = param
            bws[j] = bw

        num = np.sum((new_XB - XB)**2) / n
        den = np.sum(np.sum(new_XB, axis=1)**2)
        score = (num / den)**0.5
        XB = new_XB

        if rss_score:
            predy = np.sum(np.multiply(params, X), axis=1).reshape((-1, 1))
            new_rss = np.sum((y - predy)**2)
            score = np.abs((new_rss - rss) / new_rss)
            rss = new_rss
        scores.append(deepcopy(score))
        delta = score
        BWs.append(deepcopy(bws))

        if verbose:
            print("Current iteration:", iters, ",SOC:", np.round(score, 7))
            print("Bandwidths:", ', '.join([str(bw) for bw in bws]))

        if delta < tol:
            break

    print("iters = "+str(iters))
    opt_bws = BWs[-1]
    print("opt_bws = "+str(opt_bws))
    return (opt_bws, np.array(BWs), np.array(scores), params, err, bw_gwr)
```

##### Running the function with family = Binomial()

```python
bw_mgwbr = multi_bw(init=None,coords=coords,y=y_std, X=x_std, n=239, k=x.shape[1], family=Binomial())
```

    iters = 1
    opt_bws = [198.]

##### Running without family and offset attributes runs the normal MGWR loop

```python
bw_mgwr = multi_bw(init=None, coords=coords,y=y_std, X=x_std, n=262, k=x.shape[1])
```

    iters = 1
    opt_bws = [125.]

### Parameter check

#### Difference in parameters from the GWR - Binomial model and MGWR Binomial model

```python
(bw_mgwbr[3]==gwbr_model.params).all()
```

    True

The parameters are identical

### Bandwidths check

```python
bw_gwbr
```

    235.0

```python
bw_mgwbr[0]
```

    array([235.])

The bandwidth from both models is the same


# Logistic Regression

## Introduction
In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable $y_i$ is based on some
independent variables $\hat{x}_i$. Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters $\hat{\beta}$ to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.

Classification problems, however, are concerned with outcomes taking
the form of discrete variables (i.e. categories). We may for example,
on the basis of DNA sequencing for a number of patients, like to find
out which mutations are important for a certain disease; or based on
scans of various patients' brains, figure out if there is a tumor or
not; or given a specific physical system, we'd like to identify its
state, say whether it is an ordered or disordered system (typical
situation in solid state physics); or classify the status of a
patient, whether she/he has a stroke or not and many other similar
situations.

The most common situation we encounter when we apply logistic
regression is that of two possible outcomes, normally denoted as a
binary outcome, true or false, positive or negative, success or
failure etc.

Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters $\hat{\beta}$. The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
family of gradient descent methods. The latter are the working horses
of basically all modern machine learning algorithms.

We note also that many of the topics discussed here on logistic
regression are also commonly used in modern supervised Deep Learning
models, as we will see later.

## Basics

We consider the case where the dependent variables, also called the
responses or the outcomes, $y_i$ are discrete and only take values
from $k=0,\dots,K-1$ (i.e. $K$ classes).

The goal is to predict the
output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$
made of $n$ samples, each of which carries $p$ features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.

Let us specialize to the case of two classes only, with outputs
$y_i=0$ and $y_i=1$. Our outcomes could represent the status of a
credit card user that could default or not on her/his credit card
debt. That is

$$
y_i = \begin{bmatrix} 0 & \mathrm{no}\\  1 & \mathrm{yes} \end{bmatrix}.
$$

Before moving to the logistic model, let us try to use our linear
regression model to classify these two outcomes. We could for example
fit a linear model to the default case if $y_i > 0.5$ and the no
default case $y_i \leq 0.5$.

We would then have our
weighted linear combination, namely

$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} +  \hat{\epsilon},
\label{_auto1} \tag{1}
\end{equation}
$$

where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our
$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors.

The main problem with our function is that it takes values on the
entire real axis. In the case of logistic regression, however, the
labels $y_i$ are discrete variables. A typical example is the credit
card data discussed below here, where we can set the state of
defaulting the debt to $y_i=1$ and not to $y_i=0$ for one the persons
in the data set (see the full example below).

One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to values $\{0,1\}$,
$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise.
We will encounter this model in our first demonstration of neural networks. Historically it is called the "perceptron" model in the machine learning
literature. This model is extremely simple. However, in many cases it is more
favorable to use a ``soft" classifier that outputs
the probability of a given category. This leads us to the logistic function.

## The logistic function

The perceptron is an example of a ``hard classification" model. We
will encounter this model when we discuss neural networks as
well. Each datapoint is deterministically assigned to a category (i.e
$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a "soft"
classifier that outputs the probability of a given category rather
than a single value. For example, given $x_i$, the classifier
outputs the probability of being in a category $k$.  Logistic regression
is the most common example of a so-called soft classifier. In logistic
regression, the probability that a data point $x_i$
belongs to a category $y_i=\{0,1\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,

$$
p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
$$

Note that $1-p(t)= p(-t)$.

The following code plots the logistic function, the step function and other functions we will encounter from here and on.

```
%matplotlib inline

"""The sigmoid function (or the logistic curve) is a
function that takes any real number, z, and outputs a number (0,1).
It is useful in neural networks for assigning weights on a relative scale.
The value z is the weighted sum of parameters involved in the learning algorithm."""

import numpy
import matplotlib.pyplot as plt
import math as mt

z = numpy.arange(-5, 5, .1)
sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
sigma = sigma_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, sigma)
ax.set_ylim([-0.1, 1.1])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('sigmoid function')

plt.show()

"""Step Function"""
z = numpy.arange(-5, 5, .02)
step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
step = step_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, step)
ax.set_ylim([-0.5, 1.5])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('step function')

plt.show()

"""tanh Function"""
z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
t = numpy.tanh(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, t)
ax.set_ylim([-1.0, 1.0])
ax.set_xlim([-2*mt.pi,2*mt.pi])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('tanh function')

plt.show()
```

## Two parameters

We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities

$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
\end{align*}
$$

where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.

Note that we used

$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
$$

## Maximum likelihood

In order to define the total likelihood for all possible outcomes from a
dataset $\mathcal{D}=\{(y_i,x_i)\}$, with the binary labels
$y_i\in\{0,1\}$ and where the data points are drawn independently, we use the so-called [Maximum Likelihood Estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) principle.
We aim thus at maximizing
the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is

$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$

from which we obtain the log-likelihood and our **cost/loss** function

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
$$

Reordering the logarithms, we can rewrite the **cost/loss** function as

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\beta$.
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that

$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.

The cross entropy is a convex function of the weights $\hat{\beta}$ and,
therefore, any local minimizer is a global minimizer.

Minimizing this
cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obtain

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$

and

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$

Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an
$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a
vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first
derivative of cost function as

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
$$

If we in addition define a diagonal matrix $\hat{W}$ with elements
$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as

$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
$$

Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors

$$
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$

Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to

$$
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$

## Including more classes

Till now we have mainly focused on two classes, the so-called binary
system. Suppose we wish to extend to $K$ classes.  Let us for the sake
of simplicity assume we have only two predictors. We have then
following model

1
5

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
$$

and so on till the class $C=K-1$ class

$$
\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
$$

and the model is specified in term of $K-1$ so-called log-odds or
**logit** transformations.

In our discussion of neural networks we will encounter the above again
in terms of a slightly modified function, the so-called **Softmax** function.

The softmax function is used in various multiclass classification
methods, such as multinomial logistic regression (also known as
softmax regression), multiclass linear discriminant analysis, naive
Bayes classifiers, and artificial neural networks.  Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of $K$ distinct linear functions,
and the predicted probability for the $k$-th class given a sample
vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two
predictors):

$$
p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
$$

It is easy to extend to more predictors. The final class is

$$
p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
$$

and they sum to one. Our earlier discussions were all specialized to
the case with two classes only. It is easy to see from the above that
what we derived earlier is compatible with these equations.

To find the optimal parameters we would typically use a gradient
descent method.  Newton's method and gradient descent methods are
discussed in the material on [optimization
methods](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html).


# Plane vector equation

> # normal vector $ \vec{n} \iff $ 평면에 수직인 vector
>> ## $\vec{n} = (n_x,n_y,n_z) $ 과 절대 평행하면 안된다.
>>> ## normal vector 인($(n_x\:t, n_y\:t,n_z\:t)$) 과 평행하면 직선은 평면위에 있는 직선이 아니어야 한다. 그래서 0 이된다.
k
>>> ## $ n_x(x - x_0) + n_y(y - y_0) + n_z(z - z_0) = 0 $

```python
import sympy as sm
import sympy.vector
import matplotlib.pyplot as plt
import numpy as np

C = sm.vector.CoordSys3D('')

# noarmal vector
n = C.x*C.i + C.y*C.j + C.z*C.k
# point vector
p = n
# plane equation
n = n.subs({C.x:1,C.y:2,C.z:3})
p = n.subs({C.x:4,C.y:3,C.z:3})
C.name_scalars()
```

# Line vector equation
> ## direction vector $\vec{d} = (d_x, d_y , d_z)$
>> ## $\vec{d} = (d_x,d_y,d_z)$ 과 반드시 평행해야 함으로 각각의 방향에 대한 미분계수가 같은 비율이어야 한다.
>>> ## $
\frac{(x-x_0)}{\vec{d_x}} = \frac{(y-y_0)}{\vec{d_y}} = \frac{z-z_0}{\vec{d_z}}
$

```python

```


# Shock Tube

> WORK IN PROGRESS !!!

In this example we are going to solve the Euler equations for a one-dimensional shock tube.

\begin{equation}
    \begin{array}{c}
        \rho_t + (\rho u)_x = 0 \\
        (\rho u)_t + (\rho u u + p)_x = 0 \\
        (\rho e)_t + (u ( \rho e + p ))_x = 0 \\
    \end{array}
\end{equation}

The inputs to the network will be the independent variables $x$ and $t$ and the outputs will be the density, $\rho$, the velocity $u$ and the pressure $p$. We can compute the internal energy with the expression $\rho e = 0.5 \rho u u + p/(\gamma-1)$ with $\gamma=1.4$ for air.

```
# autoreload nangs
%reload_ext autoreload
%autoreload 2
%matplotlib inline
```

```
# imports
import math
import numpy as np
import matplotlib.pyplot as plt
import torch
```

## Define PDE

First we define our PDE and set the values for training.

```
# imports
from nangs.pde import PDE
from nangs.bocos import PeriodicBoco, DirichletBoco
from nangs.solutions import MLP

# define our PDE
class MyPDE(PDE):
    def __init__(self, inputs, outputs, params):
        super().__init__(inputs, outputs, params)
    # the loss of our NN is the PDE !
    def computePDELoss(self, grads, inputs, outputs, params):
        drdt = grads['r']['t']
        #dudt = grads['u']['t']
        #dpdt = grads['p']['t']

        r, u, p = outputs['r'], outputs['u'], outputs['p']
        ru = r*u
        drudx = self.computeGrad(ru, 'x')
        drudt = self.computeGrad(ru, 't')
        ruup = r*u*u + p
        druupdx =  self.computeGrad(ruup, 'x')
        re = p/(params['g']-1) + 0.5*r*u*u
        dredt = self.computeGrad(re, 't')
        repu = (re+p)*u
        drepudx = self.computeGrad(repu, 'x')

        return [
            drdt + drudx,
            drudt + druupdx,
            dredt + drepudx
        ]

# instanciate pde with keys for inputs/outputs
pde = MyPDE(inputs=['x', 't'], outputs=['r', 'u', 'p'], params=['g'])

# define input values for training
x = np.linspace(0,1,200)
t = np.linspace(0,0.0001,5)
g = 1.4

pde.setValues({'x': x, 't': t, 'g': np.array([g])})

# define input values for validation
x_v = np.linspace(0,1,25)
t_v = np.linspace(0,1,15)
pde.setValues({'x': x_v, 't': t_v}, train=False)
```

```
# initial condition (dirichlet for temporal dimension)
rl, pl, rr, pr = 10, 10/g, 1, 1/g
r0 = np.zeros(len(x))
u0 = np.zeros(len(x))
p0 = np.zeros(len(x))
for i, _x in enumerate(x):
    u0[i] = 0.
    if _x < 0.5:
        r0[i] = rl
        p0[i] = pl
    else:
        r0[i] = rr
        p0[i] = pr

boco = DirichletBoco('initial_condition', {'x': x, 't': np.array([0])}, {'r': r0, 'u': u0, 'p': p0})
pde.addBoco(boco)

# dirichlet b.c for the space dimension
x1, x2 = np.array([0]), np.array([1])
ones = np.ones(len(t))
boco = DirichletBoco('boco_left', {'x': x1, 't': t}, {'r': r0[0]*ones, 'u': u0[0]*ones, 'p': p0[0]*ones})
pde.addBoco(boco)
boco = DirichletBoco('boco_rigth', {'x': x2, 't': t}, {'r': r0[len(x)-1]*ones, 'u': u0[len(x)-1]*ones, 'p': p0[len(x)-1]*ones})
pde.addBoco(boco)
```

Now we define a topology for our solution and set the training parameters. Then we can find a solution for our PDE.

```
# define solution topology
mlp = MLP(pde.n_inputs, pde.n_outputs, 2, 10)
optimizer = torch.optim.Adam(mlp.parameters(), lr=0.01)
pde.compile(mlp, optimizer)
```

```
# pre-train on initial conditions

if False:
    hist = pde.warm(epochs=1000, bocos=['initial_condition'])

    for h in hist:
        plt.plot(hist[h], label=h)
    plt.legend()
    plt.grid()
    plt.yscale("log")
    plt.show()
```

```
# visualize initial condition
#pde.evaluate({'x': x, 't': np.array([0])})

plt.plot(x, r0)
plt.plot(x, u0)
plt.plot(x, p0)
#plt.plot(x, pde.outputs['r'])
#plt.plot(x, pde.outputs['u'])
#plt.plot(x, pde.outputs['p'])
plt.legend(['$\\rho$', 'u', 'p'])#, 'NN $\\rho$', 'NN u', 'NN p'])
plt.grid()
plt.xlabel('x')
plt.show()
```

```
# find the solution
hist = pde.solve(epochs=100)
```

Epoch 1/100 Losses 27.13701
 PDE  [ 0.01137 1.84210 0.04406 ]
 initial_condition 9.76178
 boco_left 10.81910
 boco_rigth 5.92363
 Val [ 0.01416 3.54421 0.01024 ]
Epoch 2/100 Losses 13.50456
 PDE  [ 0.03875 4.21175 0.03691 ]
 initial_condition 6.20692
 boco_left 3.88459
 boco_rigth 1.98391
 Val [ 0.13403 3.04941 0.37196 ]
Epoch 3/100 Losses 7.70895
 PDE  [ 0.11006 2.17749 0.16551 ]
 initial_condition 4.63856
 boco_left 1.56801
 boco_rigth 0.68470
 Val [ 0.02772 4.10926 0.00200 ]
Epoch 4/100 Losses 6.13387
 PDE  [ 0.02257 2.36967 0.01361 ]
 initial_condition 4.20727
 boco_left 0.44635
 boco_rigth 0.67830
 Val [ 0.01792 3.40029 0.00213 ]
Epoch 5/100 Losses 6.17765
 PDE  [ 0.01699 2.38564 0.00508 ]
 initial_condition 4.25539
 boco_left 0.54191
 boco_rigth 0.57778
 Val [ 0.04201 3.63548 0.00500 ]
Epoch 6/100 Losses 6.23349
 PDE  [ 0.02880 2.30698 0.00936 ]
 initial_condition 4.31495
 boco_left 0.52889
 boco_rigth 0.60793
 Val [ 0.03126 3.71874 0.00744 ]
Epoch 7/100 Losses 6.23741
 PDE  [ 0.03729 2.30446 0.00484 ]
 initial_condition 4.31246
 boco_left 0.50566
 boco_rigth 0.63710
 Val [ 0.06019 3.65287 0.00393 ]
Epoch 8/100 Losses 6.15140
 PDE  [ 0.04724 2.27922 0.00684 ]
 initial_condition 4.22936
 boco_left 0.52204
 boco_rigth 0.62224
 Val [ 0.06534 3.23061 0.03209 ]
Epoch 9/100 Losses 6.09006
 PDE  [ 0.08192 2.17616 0.04325 ]
 initial_condition 4.12011
 boco_left 0.55658
 boco_rigth 0.64627
 Val [ 0.08366 3.23414 0.00163 ]
Epoch 10/100 Losses 6.30046
 PDE  [ 0.09467 2.21366 0.03608 ]
 initial_condition 4.35207
 boco_left 0.52335
 boco_rigth 0.64356
 Val [ 0.13954 3.07101 0.00370 ]
Epoch 11/100 Losses 6.07473
 PDE  [ 0.11000 2.04462 0.01569 ]
 initial_condition 4.11861
 boco_left 0.56722
 boco_rigth 0.66546
 Val [ 0.20993 3.08170 0.01288 ]
Epoch 12/100 Losses 6.22225
 PDE  [ 0.15957 2.10611 0.07278 ]
 initial_condition 4.23111
 boco_left 0.55935
 boco_rigth 0.65231
 Val [ 0.21513 3.01365 0.04033 ]
Epoch 13/100 Losses 6.26795
 PDE  [ 0.15900 1.93901 0.01253 ]
 initial_condition 4.30450
 boco_left 0.59305
 boco_rigth 0.66689
 Val [ 0.43075 3.21585 0.15985 ]
Epoch 14/100 Losses 6.05784
 PDE  [ 0.21222 2.02416 0.04818 ]
 initial_condition 4.07822
 boco_left 0.54743
 boco_rigth 0.67067
 Val [ 0.36150 2.98084 0.05435 ]
Epoch 15/100 Losses 6.00555
 PDE  [ 0.23019 1.92777 0.03399 ]
 initial_condition 4.06894
 boco_left 0.56891
 boco_rigth 0.63705
 Val [ 0.45294 2.75322 0.07711 ]
Epoch 16/100 Losses 6.03611
 PDE  [ 0.28556 1.79174 0.07033 ]
 initial_condition 4.08759
 boco_left 0.57899
 boco_rigth 0.65365
 Val [ 0.44040 2.85138 0.01437 ]
Epoch 17/100 Losses 6.25810
 PDE  [ 0.32114 1.77717 0.04566 ]
 initial_condition 4.30781
 boco_left 0.54478
 boco_rigth 0.69085
 Val [ 0.36701 2.56328 0.01309 ]
Epoch 18/100 Losses 6.21947
 PDE  [ 0.33414 1.70629 0.02911 ]
 initial_condition 4.32516
 boco_left 0.53339
 boco_rigth 0.67107
 Val [ 0.43984 2.62426 0.02825 ]
Epoch 19/100 Losses 6.03607
 PDE  [ 0.38832 1.69083 0.09358 ]
 initial_condition 4.10378
 boco_left 0.58692
 boco_rigth 0.62113
 Val [ 0.65631 2.62864 0.05882 ]
Epoch 20/100 Losses 6.33139
 PDE  [ 0.41916 1.57457 0.18539 ]
 initial_condition 4.35224
 boco_left 0.61983
 boco_rigth 0.63295
 Val [ 0.58464 3.13858 0.28228 ]
Epoch 21/100 Losses 6.07992
 PDE  [ 0.41814 1.65240 0.06626 ]
 initial_condition 4.17719
 boco_left 0.57437
 boco_rigth 0.61610
 Val [ 0.66800 2.49661 0.02627 ]
Epoch 22/100 Losses 5.86811
 PDE  [ 0.43748 1.60104 0.02096 ]
 initial_condition 4.01572
 boco_left 0.55117
 boco_rigth 0.61473
 Val [ 0.67713 2.96567 0.01590 ]
Epoch 23/100 Losses 6.14989
 PDE  [ 0.46029 1.53705 0.01700 ]
 initial_condition 4.30856
 boco_left 0.55673
 boco_rigth 0.61315
 Val [ 0.68574 2.57141 0.11994 ]
Epoch 24/100 Losses 6.20892
 PDE  [ 0.49934 1.48840 0.04788 ]
 initial_condition 4.36774
 boco_left 0.59481
 boco_rigth 0.56784
 Val [ 0.68741 2.62034 0.00678 ]
Epoch 25/100 Losses 5.94031
 PDE  [ 0.52869 1.53709 0.05744 ]
 initial_condition 4.07011
 boco_left 0.53303
 boco_rigth 0.62944
 Val [ 0.82443 2.29832 0.23013 ]
Epoch 26/100 Losses 6.01184
 PDE  [ 0.56459 1.38055 0.18514 ]
 initial_condition 4.06185
 boco_left 0.62274
 boco_rigth 0.61715
 Val [ 0.86264 2.17988 0.06996 ]
Epoch 27/100 Losses 6.19123
 PDE  [ 0.55528 1.46252 0.07295 ]
 initial_condition 4.29480
 boco_left 0.55790
 boco_rigth 0.64160
 Val [ 0.98311 2.17906 0.05023 ]
Epoch 28/100 Losses 6.07467
 PDE  [ 0.56424 1.36135 0.03110 ]
 initial_condition 4.23999
 boco_left 0.58957
 boco_rigth 0.59288
 Val [ 0.90094 2.34489 0.03259 ]
Epoch 29/100 Losses 6.06330
 PDE  [ 0.59352 1.37705 0.04661 ]
 initial_condition 4.20991
 boco_left 0.58689
 boco_rigth 0.59411
 Val [ 0.94330 2.27223 0.02664 ]
Epoch 30/100 Losses 5.93384
 PDE  [ 0.60262 1.36132 0.06352 ]
 initial_condition 4.09820
 boco_left 0.57786
 boco_rigth 0.58195
 Val [ 0.90871 2.03516 0.10739 ]
Epoch 31/100 Losses 6.20559
 PDE  [ 0.61847 1.34441 0.03240 ]
 initial_condition 4.37693
 boco_left 0.59222
 boco_rigth 0.57135
 Val [ 1.17798 2.41029 0.09516 ]
Epoch 32/100 Losses 6.07979
 PDE  [ 0.63213 1.33885 0.06145 ]
 initial_condition 4.22218
 boco_left 0.57797
 boco_rigth 0.60216
 Val [ 1.09915 2.40287 0.02163 ]
Epoch 33/100 Losses 5.85205
 PDE  [ 0.64817 1.30459 0.02471 ]
 initial_condition 4.04956
 boco_left 0.57375
 boco_rigth 0.56957
 Val [ 1.08262 2.32288 0.02996 ]
Epoch 34/100 Losses 5.95815
 PDE  [ 0.65441 1.23722 0.03658 ]
 initial_condition 4.14300
 boco_left 0.63584
 boco_rigth 0.53657
 Val [ 1.06263 2.40656 0.07999 ]
Epoch 35/100 Losses 5.97141
 PDE  [ 0.68474 1.29634 0.05431 ]
 initial_condition 4.13304
 boco_left 0.60479
 boco_rigth 0.55510
 Val [ 0.97348 1.81314 0.00667 ]
Epoch 36/100 Losses 6.13389
 PDE  [ 0.72683 1.26676 0.19346 ]
 initial_condition 4.22662
 boco_left 0.62409
 boco_rigth 0.55416
 Val [ 0.88758 2.06664 0.17120 ]
Epoch 37/100 Losses 6.07879
 PDE  [ 0.66667 1.23009 0.08620 ]
 initial_condition 4.21563
 boco_left 0.62277
 boco_rigth 0.57941
 Val [ 1.09486 2.00067 0.04023 ]
Epoch 38/100 Losses 6.00300
 PDE  [ 0.68731 1.25192 0.03960 ]
 initial_condition 4.16646
 boco_left 0.59508
 boco_rigth 0.58185
 Val [ 1.10750 1.82361 0.01893 ]
Epoch 39/100 Losses 5.87361
 PDE  [ 0.69670 1.25524 0.01388 ]
 initial_condition 4.05850
 boco_left 0.58797
 boco_rigth 0.57187
 Val [ 1.12263 2.29792 0.01326 ]
Epoch 40/100 Losses 6.03465
 PDE  [ 0.69538 1.22784 0.01135 ]
 initial_condition 4.19962
 boco_left 0.62030
 boco_rigth 0.56987
 Val [ 1.14050 1.90212 0.02050 ]
Epoch 41/100 Losses 6.02729
 PDE  [ 0.69482 1.16691 0.00764 ]
 initial_condition 4.22404
 boco_left 0.60829
 boco_rigth 0.57185
 Val [ 1.11697 1.99970 0.00767 ]
Epoch 42/100 Losses 6.01550
 PDE  [ 0.72469 1.21222 0.00940 ]
 initial_condition 4.21882
 boco_left 0.59861
 boco_rigth 0.54931
 Val [ 1.11941 1.88368 0.01356 ]
Epoch 43/100 Losses 6.08522
 PDE  [ 0.75649 1.18843 0.16467 ]
 initial_condition 4.19221
 boco_left 0.61160
 boco_rigth 0.57823
 Val [ 1.31298 1.65426 0.28242 ]
Epoch 44/100 Losses 6.13264
 PDE  [ 0.73085 1.21233 0.11708 ]
 initial_condition 4.27131
 boco_left 0.63572
 boco_rigth 0.53886
 Val [ 1.25092 1.79224 0.21148 ]
Epoch 45/100 Losses 6.18026
 PDE  [ 0.74808 1.21432 0.23152 ]
 initial_condition 4.25003
 boco_left 0.64309
 boco_rigth 0.55583
 Val [ 1.28790 2.06035 0.03745 ]
Epoch 46/100 Losses 6.01155
 PDE  [ 0.70454 1.21035 0.03636 ]
 initial_condition 4.19721
 boco_left 0.60961
 boco_rigth 0.55432
 Val [ 1.17070 1.96635 0.01274 ]
Epoch 47/100 Losses 6.06230
 PDE  [ 0.71057 1.15574 0.01172 ]
 initial_condition 4.24467
 boco_left 0.61166
 boco_rigth 0.57996
 Val [ 1.21648 2.37247 0.02529 ]
Epoch 48/100 Losses 5.92021
 PDE  [ 0.74544 1.27197 0.01100 ]
 initial_condition 4.10973
 boco_left 0.56904
 boco_rigth 0.56531
 Val [ 1.13130 1.79277 0.00730 ]
Epoch 49/100 Losses 5.95317
 PDE  [ 0.74365 1.15752 0.01540 ]
 initial_condition 4.11034
 boco_left 0.64558
 boco_rigth 0.55839
 Val [ 1.12551 1.76950 0.01638 ]
Epoch 50/100 Losses 5.85695
 PDE  [ 0.73996 1.17378 0.01132 ]
 initial_condition 4.04243
 boco_left 0.60745
 boco_rigth 0.56538
 Val [ 1.13942 1.86700 0.00546 ]
Epoch 51/100 Losses 6.02050
 PDE  [ 0.74235 1.15224 0.03220 ]
 initial_condition 4.19389
 boco_left 0.59049
 boco_rigth 0.59386
 Val [ 1.15168 1.74460 0.03799 ]
Epoch 52/100 Losses 5.97718
 PDE  [ 0.77362 1.16215 0.12186 ]
 initial_condition 4.06816
 boco_left 0.61701
 boco_rigth 0.60613
 Val [ 1.10424 1.67282 0.06586 ]
Epoch 53/100 Losses 6.00150
 PDE  [ 0.75857 1.16231 0.03768 ]
 initial_condition 4.16726
 boco_left 0.61283
 boco_rigth 0.56855
 Val [ 1.16654 1.64218 0.01349 ]
Epoch 54/100 Losses 6.04009
 PDE  [ 0.73924 1.13837 0.00786 ]
 initial_condition 4.22674
 boco_left 0.60006
 boco_rigth 0.58480
 Val [ 1.22837 1.98844 0.01332 ]
Epoch 55/100 Losses 5.94878
 PDE  [ 0.75016 1.14856 0.01056 ]
 initial_condition 4.12936
 boco_left 0.60805
 boco_rigth 0.57495
 Val [ 1.22595 2.09372 0.11286 ]
Epoch 56/100 Losses 5.97244
 PDE  [ 0.75289 1.12988 0.04051 ]
 initial_condition 4.16364
 boco_left 0.61048
 boco_rigth 0.55723
 Val [ 1.29808 1.94787 0.01776 ]
Epoch 57/100 Losses 5.98113
 PDE  [ 0.77105 1.14715 0.01453 ]
 initial_condition 4.16579
 boco_left 0.61184
 boco_rigth 0.55926
 Val [ 1.13418 1.71097 0.06672 ]
Epoch 58/100 Losses 6.06930
 PDE  [ 0.74841 1.10264 0.01861 ]
 initial_condition 4.25312
 boco_left 0.60802
 boco_rigth 0.58495
 Val [ 1.23462 1.86010 0.01072 ]
Epoch 59/100 Losses 5.95179
 PDE  [ 0.74159 1.09642 0.00929 ]
 initial_condition 4.15142
 boco_left 0.59880
 boco_rigth 0.58581
 Val [ 1.25817 1.90375 0.01902 ]
Epoch 60/100 Losses 5.93595
 PDE  [ 0.77557 1.13287 0.00940 ]
 initial_condition 4.13370
 boco_left 0.61828
 boco_rigth 0.54470
 Val [ 1.27884 2.06606 0.02109 ]
Epoch 61/100 Losses 6.13273
 PDE  [ 0.78919 1.14066 0.05896 ]
 initial_condition 4.30921
 boco_left 0.61616
 boco_rigth 0.54441
 Val [ 1.17534 2.00432 0.00301 ]
Epoch 62/100 Losses 6.08177
 PDE  [ 0.75261 1.11972 0.02173 ]
 initial_condition 4.25404
 boco_left 0.61314
 boco_rigth 0.58324
 Val [ 1.15677 1.65373 0.00615 ]
Epoch 63/100 Losses 5.87341
 PDE  [ 0.78337 1.13400 0.00723 ]
 initial_condition 4.06855
 boco_left 0.62609
 boco_rigth 0.53724
 Val [ 1.26401 1.96568 0.00803 ]
Epoch 64/100 Losses 5.99461
 PDE  [ 0.77407 1.12434 0.00729 ]
 initial_condition 4.18438
 boco_left 0.62502
 boco_rigth 0.54998
 Val [ 1.22559 1.73108 0.02020 ]
Epoch 65/100 Losses 6.09383
 PDE  [ 0.77906 1.09442 0.18308 ]
 initial_condition 4.19320
 boco_left 0.64204
 boco_rigth 0.57308
 Val [ 1.47987 1.76207 0.09509 ]
Epoch 66/100 Losses 5.96886
 PDE  [ 0.77545 1.14614 0.09549 ]
 initial_condition 4.13293
 boco_left 0.63134
 boco_rigth 0.53224
 Val [ 1.13855 1.73040 0.04087 ]
Epoch 67/100 Losses 6.14937
 PDE  [ 0.75872 1.14339 0.01834 ]
 initial_condition 4.32833
 boco_left 0.61260
 boco_rigth 0.56829
 Val [ 1.17051 1.86167 0.00919 ]
Epoch 68/100 Losses 5.79506
 PDE  [ 0.75398 1.13270 0.00729 ]
 initial_condition 4.00252
 boco_left 0.58467
 boco_rigth 0.57654
 Val [ 1.23927 1.98467 0.01005 ]
Epoch 69/100 Losses 6.14756
 PDE  [ 0.74914 1.09518 0.00837 ]
 initial_condition 4.33848
 boco_left 0.62812
 boco_rigth 0.56340
 Val [ 1.23656 1.96225 0.01011 ]
Epoch 70/100 Losses 5.98912
 PDE  [ 0.79396 1.17716 0.02065 ]
 initial_condition 4.17119
 boco_left 0.60031
 boco_rigth 0.55371
 Val [ 1.10748 2.05827 0.50840 ]
Epoch 71/100 Losses 6.19440
 PDE  [ 0.75904 1.08837 0.13214 ]
 initial_condition 4.31399
 boco_left 0.64875
 boco_rigth 0.57181
 Val [ 1.08860 1.72124 0.00559 ]
Epoch 72/100 Losses 5.97154
 PDE  [ 0.74823 1.12597 0.06335 ]
 initial_condition 4.14165
 boco_left 0.62730
 boco_rigth 0.55675
 Val [ 1.31651 1.94861 0.02031 ]
Epoch 73/100 Losses 5.95960
 PDE  [ 0.77592 1.14655 0.01323 ]
 initial_condition 4.16221
 boco_left 0.59839
 boco_rigth 0.55376
 Val [ 1.19258 1.86544 0.01528 ]
Epoch 74/100 Losses 5.99206
 PDE  [ 0.77681 1.15991 0.01542 ]
 initial_condition 4.16615
 boco_left 0.60071
 boco_rigth 0.57449
 Val [ 1.22937 1.83967 0.02706 ]
Epoch 75/100 Losses 6.02178
 PDE  [ 0.75403 1.10702 0.01449 ]
 initial_condition 4.18850
 boco_left 0.63038
 boco_rigth 0.57773
 Val [ 1.20436 1.63989 0.01509 ]
Epoch 76/100 Losses 6.17608
 PDE  [ 0.76889 1.10342 0.00959 ]
 initial_condition 4.35407
 boco_left 0.63393
 boco_rigth 0.56078
 Val [ 1.31302 1.89737 0.09095 ]
Epoch 77/100 Losses 5.91743
 PDE  [ 0.77495 1.10570 0.02848 ]
 initial_condition 4.11325
 boco_left 0.60171
 boco_rigth 0.56610
 Val [ 1.30932 1.86280 0.02476 ]
Epoch 78/100 Losses 6.00126
 PDE  [ 0.76525 1.11695 0.00775 ]
 initial_condition 4.19795
 boco_left 0.62084
 boco_rigth 0.55249
 Val [ 1.25241 1.72084 0.01576 ]
Epoch 79/100 Losses 6.09379
 PDE  [ 0.77990 1.12215 0.00937 ]
 initial_condition 4.25214
 boco_left 0.58968
 boco_rigth 0.61483
 Val [ 1.32979 2.17522 0.01309 ]
Epoch 80/100 Losses 5.97715
 PDE  [ 0.78676 1.10367 0.04513 ]
 initial_condition 4.11970
 boco_left 0.60305
 boco_rigth 0.60922
 Val [ 1.38673 1.90646 0.20594 ]
Epoch 81/100 Losses 6.01564
 PDE  [ 0.78498 1.15171 0.05586 ]
 initial_condition 4.17177
 boco_left 0.61958
 boco_rigth 0.56009
 Val [ 1.27801 1.70719 0.02258 ]
Epoch 82/100 Losses 6.08611
 PDE  [ 0.76266 1.10159 0.01222 ]
 initial_condition 4.26098
 boco_left 0.61538
 boco_rigth 0.58425
 Val [ 1.20052 1.94259 0.06863 ]
Epoch 83/100 Losses 5.97573
 PDE  [ 0.75550 1.07917 0.02419 ]
 initial_condition 4.16218
 boco_left 0.63554
 boco_rigth 0.55839
 Val [ 1.17818 1.77816 0.02890 ]
Epoch 84/100 Losses 5.87951
 PDE  [ 0.80258 1.16009 0.01366 ]
 initial_condition 4.07901
 boco_left 0.58986
 boco_rigth 0.55187
 Val [ 1.22073 1.68865 0.03455 ]
Epoch 85/100 Losses 6.11742
 PDE  [ 0.78468 1.10030 0.16482 ]
 initial_condition 4.23516
 boco_left 0.60185
 boco_rigth 0.59714
 Val [ 1.55509 1.62964 0.16589 ]
Epoch 86/100 Losses 5.98185
 PDE  [ 0.76029 1.11538 0.06804 ]
 initial_condition 4.14595
 boco_left 0.61815
 boco_rigth 0.56985
 Val [ 1.24295 1.87269 0.02082 ]
Epoch 87/100 Losses 5.98456
 PDE  [ 0.77646 1.14848 0.01403 ]
 initial_condition 4.16172
 boco_left 0.60734
 boco_rigth 0.56918
 Val [ 1.17753 1.70046 0.02231 ]
Epoch 88/100 Losses 5.94938
 PDE  [ 0.74022 1.09527 0.02175 ]
 initial_condition 4.13452
 boco_left 0.60034
 boco_rigth 0.59544
 Val [ 1.19464 1.92179 0.02809 ]
Epoch 89/100 Losses 6.00905
 PDE  [ 0.78522 1.14561 0.00945 ]
 initial_condition 4.18745
 boco_left 0.59437
 boco_rigth 0.58047
 Val [ 1.15520 1.85083 0.00865 ]
Epoch 90/100 Losses 6.19300
 PDE  [ 0.75499 1.10188 0.03987 ]
 initial_condition 4.34314
 boco_left 0.65172
 boco_rigth 0.56589
 Val [ 1.12855 2.03394 0.28350 ]
Epoch 91/100 Losses 6.11425
 PDE  [ 0.76148 1.12981 0.10519 ]
 initial_condition 4.22207
 boco_left 0.62300
 boco_rigth 0.60368
 Val [ 1.20725 2.07004 0.02634 ]
Epoch 92/100 Losses 6.00604
 PDE  [ 0.76895 1.11970 0.03229 ]
 initial_condition 4.16612
 boco_left 0.63692
 boco_rigth 0.56269
 Val [ 1.32579 2.21467 0.03298 ]
Epoch 93/100 Losses 6.01436
 PDE  [ 0.77347 1.17589 0.02036 ]
 initial_condition 4.18438
 boco_left 0.61434
 boco_rigth 0.55907
 Val [ 1.31291 1.86149 0.02758 ]
Epoch 94/100 Losses 5.83688
 PDE  [ 0.75852 1.11470 0.00977 ]
 initial_condition 4.02140
 boco_left 0.60869
 boco_rigth 0.57913
 Val [ 1.24080 1.77220 0.01125 ]
Epoch 95/100 Losses 6.10368
 PDE  [ 0.78077 1.12085 0.00796 ]
 initial_condition 4.27522
 boco_left 0.61114
 boco_rigth 0.58079
 Val [ 1.19113 1.93432 0.00207 ]
Epoch 96/100 Losses 5.97724
 PDE  [ 0.75071 1.08519 0.04109 ]
 initial_condition 4.15381
 boco_left 0.63305
 boco_rigth 0.56472
 Val [ 1.38185 1.96000 0.10136 ]
Epoch 97/100 Losses 6.02735
 PDE  [ 0.76048 1.11884 0.01921 ]
 initial_condition 4.21696
 boco_left 0.60225
 boco_rigth 0.57530
 Val [ 1.22893 1.91598 0.00940 ]
Epoch 98/100 Losses 5.95622
 PDE  [ 0.78620 1.15639 0.00906 ]
 initial_condition 4.14751
 boco_left 0.60602
 boco_rigth 0.55214
 Val [ 1.21065 1.80602 0.01657 ]
Epoch 99/100 Losses 6.04956
 PDE  [ 0.76251 1.09000 0.01412 ]
 initial_condition 4.24115
 boco_left 0.62710
 boco_rigth 0.55911
 Val [ 1.20038 1.63458 0.03065 ]
Epoch 100/100 Losses 5.99090
 PDE  [ 0.79273 1.13001 0.01056 ]
 initial_condition 4.20272
 boco_left 0.59627
 boco_rigth 0.54747
 Val [ 1.24823 1.99616 0.00724 ]

    /opt/conda/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3257: RuntimeWarning: Mean of empty slice.
      out=out, **kwargs)
    /opt/conda/lib/python3.7/site-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
      ret = ret.dtype.type(ret / rcount)

```
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))
ax1.plot(hist['train_loss'], label="train_loss")
ax1.plot(hist['val_loss'], label="val_loss")
ax1.grid(True)
ax1.legend()
ax1.set_yscale("log")
for boco in pde.bocos:
    ax2.plot(hist['bocos'][boco.name], label=boco.name)
ax2.legend()
ax2.grid(True)
ax2.set_yscale("log")
plt.show()
```

Finally, we can evaluate our solution.

```
# evaluate the solution
x = np.linspace(0,1,100)
t = np.linspace(0,0.0001,20)
r, u, p = [], [], []
for _t in t:
    pde.evaluate({'x': x, 't': np.array([_t])})
    r.append(pde.outputs['r'])
    u.append(pde.outputs['u'])
    p.append(pde.outputs['p'])

from matplotlib import animation, rc
rc('animation', html='html5')

def plot(x, r, u, p, t):
    ax.clear()
    tit = ax.set_title(f"t = {t:.4f}", fontsize=14)
    ax.plot(x, r, label="$\\rho$")
    ax.plot(x, u, label="u")
    ax.plot(x, p, label="p")
    ax.set_xlabel("x", fontsize=14)
    ax.legend(loc="upper right")
    ax.grid(True)
    ax.set_xlim([0, 1])
    ax.set_ylim([-0.2, 10.2])
    return [tit]

def get_anim(fig, ax, x, r, u, p, t):
    def anim(i):
        return plot(x, r[i], u[i], p[i], t[i])
    return anim

fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(111, autoscale_on=False)
animate = get_anim(fig, ax, x, r, u, p, t)
anim = animation.FuncAnimation(fig, animate, frames=len(t), interval=100, blit=True)
```

```
anim
```

```

```