# Linear Gaussian filtering and smoothing

Provided are two examples of linear state-space models on which one can perform Bayesian filtering and smoothing in order to obtain
a posterior distribution over a latent state trajectory based on noisy observations.
In order to understand the theory behind these methods in detail we refer to [1] and [2].

We provide examples for two different types of state-space model:
1. [Linear, Discrete State-Space Model](#1.-Linear-Discrete-State-Space-Model:-Car-Tracking): Car Tracking
2. [Linear, Continuous-Discrete State-Space Model](#2.-Linear-Continuous-Discrete-State-Space-Model:-Ornstein-Uhlenbeck-Process): The Ornstein-Uhlenbeck Process

**References**:
> [1] Särkkä, Simo, and Solin, Arno. Applied Stochastic Differential Equations. Cambridge University Press, 2019.
>
> [2] Särkkä, Simo. Bayesian Filtering and Smoothing. Cambridge University Press, 2013.

```python
import numpy as np

import probnum as pn
from probnum import filtsmooth, randvars, randprocs
from probnum.problems import TimeSeriesRegressionProblem
```

```python
rng = np.random.default_rng(seed=123)
```

```python
# Make inline plots vector graphics instead of raster graphics
%matplotlib inline
from IPython.display import set_matplotlib_formats

set_matplotlib_formats("pdf", "svg")

# Plotting
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec

plt.style.use("../../probnum.mplstyle")
```

    /tmp/ipykernel_97997/236124620.py:5: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`
      set_matplotlib_formats("pdf", "svg")

## 1. **Linear Discrete** State-Space Model: Car Tracking

---

We begin showcasing the arguably most simple case in which we consider the following state-space model. Consider matrices $A \in \mathbb{R}^{d \times d}$ and $H \in \mathbb{R}^{m \times d}$ where $d$ is the state dimension and $m$ is the dimension of the measurements. Then we define the dynamics and the measurement model as follows:

For $k = 1, \dots, K$ and $x_0 \sim \mathcal{N}(\mu_0, \Sigma_0)$:

$$
\begin{align}
    \boldsymbol{x}_k &\sim \mathcal{N}(\boldsymbol{A} \, \boldsymbol{x}_{k-1}, \boldsymbol{Q}) \\
    \boldsymbol{y}_k &\sim \mathcal{N}(\boldsymbol{H} \, \boldsymbol{x}_k, \boldsymbol{R})
\end{align}
$$

This defines a dynamics model that assumes a state $\boldsymbol{x}_k$ in a **discrete** sequence of states arising from a linear projection of the previous state $x_{k-1}$ corrupted with additive Gaussian noise under a **process noise** covariance matrix $Q$.
Similarly, the measurements $\boldsymbol{y}_k$ are assumed to be linear projections of the latent state under additive Gaussian noise according to a **measurement noise** covariance $R$.
In the following example we consider projections and covariances that are constant over the state and measurement trajectories (linear time invariant, or **LTI**). Note that this can be generalized to a linear time-varying state-space model, as well. Then $A$ is a function $A: \mathbb{T} \rightarrow \mathbb{R}^{d \times d}$ and $H$ is a function $H: \mathbb{T} \rightarrow \mathbb{R}^{m \times d}$ where $\mathbb{T}$ is the "time dimension".

In other words, here, every relationship is linear and every distribution is a Gaussian distribution.
Under these simplifying assumptions it is possible to obtain a filtering posterior distribution over the state trajectory $(\boldsymbol{x}_k)_{k=1}^{K}$ by using a
**Kalman Filter**. The example is taken from Example 3.6 in [2].

### Define State-Space Model

#### I. Discrete Dynamics Model: Linear, Time-Invariant, Gaussian Transitions

```python
state_dim = 4
observation_dim = 2
```

```python
delta_t = 0.2
# Define linear transition operator
dynamics_transition_matrix = np.eye(state_dim) + delta_t * np.diag(np.ones(2), 2)
# Define process noise (covariance) matrix
noise_matrix = (
    np.diag(np.array([delta_t ** 3 / 3, delta_t ** 3 / 3, delta_t, delta_t]))
    + np.diag(np.array([delta_t ** 2 / 2, delta_t ** 2 / 2]), 2)
    + np.diag(np.array([delta_t ** 2 / 2, delta_t ** 2 / 2]), -2)
)
```

To create a discrete, LTI Gaussian dynamics model, `probnum` provides the `LTIGaussian` class that takes
- `state_trans_mat` : the linear transition matrix (above: $A$)
- `shift_vec` : a force vector for _affine_ transformations of the state (here: zero)
- `proc_noise_cov_mat` : the covariance matrix for the Gaussian process noise

```python
# Create discrete, Linear Time-Invariant Gaussian dynamics model
noise = randvars.Normal(mean=np.zeros(state_dim), cov=noise_matrix)
dynamics_model = randprocs.markov.discrete.LTIGaussian(
    transition_matrix=dynamics_transition_matrix,
    noise=noise,
)
```

#### II. Discrete Measurement Model: Linear, Time-Invariant, Gaussian Measurements

```python
measurement_marginal_variance = 0.5
measurement_matrix = np.eye(observation_dim, state_dim)
measurement_noise_matrix = measurement_marginal_variance * np.eye(observation_dim)
```

```python
noise = randvars.Normal(mean=np.zeros(observation_dim), cov=measurement_noise_matrix)
measurement_model = randprocs.markov.discrete.LTIGaussian(
    transition_matrix=measurement_matrix,
    noise=noise,
)
```

#### III. Initial State Random Variable

```python
mu_0 = np.zeros(state_dim)
sigma_0 = 0.5 * measurement_marginal_variance * np.eye(state_dim)
initial_state_rv = randvars.Normal(mean=mu_0, cov=sigma_0)
```

```python
prior_process = randprocs.markov.MarkovProcess(
    transition=dynamics_model, initrv=initial_state_rv, initarg=0.0
)
```

### Generate Data for the State-Space Model

`statespace.generate_artificial_measurements()` is used to sample both latent states and noisy observations from the specified state space model.

```python
time_grid = np.arange(0.0, 10.0, step=delta_t)
```

```python
latent_states, observations = randprocs.markov.utils.generate_artificial_measurements(
    rng=rng,
    prior_process=prior_process,
    measmod=measurement_model,
    times=time_grid,
)
```

```python
regression_problem = TimeSeriesRegressionProblem(
    observations=observations,
    locations=time_grid,
    measurement_models=[measurement_model] * len(time_grid),
)
```

### Kalman Filtering

#### I. Kalman Filter

```python
kalman_filter = filtsmooth.gaussian.Kalman(prior_process)
```

#### II. Perform Kalman Filtering + Rauch-Tung-Striebel Smoothing

```python
state_posterior, _ = kalman_filter.filtsmooth(regression_problem)
```

The method `filtsmooth` returns a `KalmanPosterior` object which provides convenience functions for e.g. sampling and interpolation.
We can also extract the just computed posterior smoothing state variables by querying the `.state_rvs` property.
This yields a list of Gaussian Random Variables from which we can extract the statistics in order to visualize them.

```python
grid = state_posterior.locations
posterior_state_rvs = (
    state_posterior.states
)  # List of <num_time_points> Normal Random Variables
posterior_state_means = posterior_state_rvs.mean  # Shape: (num_time_points, state_dim)
posterior_state_covs = (
    posterior_state_rvs.cov
)  # Shape: (num_time_points, state_dim, state_dim)
```

### Visualize Results

```python
state_fig = plt.figure()
state_fig_gs = gridspec.GridSpec(ncols=2, nrows=2, figure=state_fig)

ax_00 = state_fig.add_subplot(state_fig_gs[0, 0])
ax_01 = state_fig.add_subplot(state_fig_gs[0, 1])
ax_10 = state_fig.add_subplot(state_fig_gs[1, 0])
ax_11 = state_fig.add_subplot(state_fig_gs[1, 1])

# Plot means
mu_x_1, mu_x_2, mu_x_3, mu_x_4 = [posterior_state_means[:, i] for i in range(state_dim)]

ax_00.plot(grid, mu_x_1, label="posterior mean")
ax_01.plot(grid, mu_x_2)
ax_10.plot(grid, mu_x_3)
ax_11.plot(grid, mu_x_4)

# Plot marginal standard deviations
std_x_1, std_x_2, std_x_3, std_x_4 = [
    np.sqrt(posterior_state_covs[:, i, i]) for i in range(state_dim)
]

ax_00.fill_between(
    grid,
    mu_x_1 - 1.96 * std_x_1,
    mu_x_1 + 1.96 * std_x_1,
    alpha=0.2,
    label="1.96 marginal stddev",
)
ax_01.fill_between(grid, mu_x_2 - 1.96 * std_x_2, mu_x_2 + 1.96 * std_x_2, alpha=0.2)
ax_10.fill_between(grid, mu_x_3 - 1.96 * std_x_3, mu_x_3 + 1.96 * std_x_3, alpha=0.2)
ax_11.fill_between(grid, mu_x_4 - 1.96 * std_x_4, mu_x_4 + 1.96 * std_x_4, alpha=0.2)

# Plot groundtruth
obs_x_1, obs_x_2 = [observations[:, i] for i in range(observation_dim)]

ax_00.scatter(time_grid, obs_x_1, marker=".", label="measurements")
ax_01.scatter(time_grid, obs_x_2, marker=".")

# Add labels etc.
ax_00.set_xlabel("t")
ax_01.set_xlabel("t")
ax_10.set_xlabel("t")
ax_11.set_xlabel("t")

ax_00.set_title(r"$x_1$")
ax_01.set_title(r"$x_2$")
ax_10.set_title(r"$x_3$")
ax_11.set_title(r"$x_4$")
handles, labels = ax_00.get_legend_handles_labels()
state_fig.legend(handles, labels, loc="center left", bbox_to_anchor=(1, 0.5))

state_fig.tight_layout()
```

## 2. **Linear Continuous-Discrete** State-Space Model: Ornstein-Uhlenbeck Process

---

Now, consider we have a look at **continuous** dynamics. We assume that there is a continuous process that defines the dynamics of our latent space from which we collect discrete linear-Gaussian measurements (as above). Only the dynamics model becomes continuous. In particular, we formulate the dynamics as a stochastic process in terms of a linear time-invariant stochastic differential equation (LTISDE). We refer to [1] for more details.
Consider matrices $\boldsymbol{F} \in \mathbb{R}^{d \times d}$, $\boldsymbol{L} \in \mathbb{R}^{s \times d}$ and $H \in \mathbb{R}^{m \times d}$ where $d$ is the state dimension and $m$ is the dimension of the measurements.
We define the following **continuous-discrete** state-space model:

Let $x(t_0) \sim \mathcal{N}(\mu_0, \Sigma_0)$.

$$
\begin{align}
    d\boldsymbol{x} &= \boldsymbol{F} \, \boldsymbol{x} \, dt + \boldsymbol{L} \, d \boldsymbol{\omega} \\
    \boldsymbol{y}_k &\sim \mathcal{N}(\boldsymbol{H} \, \boldsymbol{x}(t_k), \boldsymbol{R}), \qquad k = 1, \dots, K
\end{align}
$$

where $\boldsymbol{\omega} \in \mathbb{R}^s$ denotes a vector of driving forces (often Brownian Motion).

Note that this can be generalized to a linear time-varying state-space model, as well. Then $\boldsymbol{F}$ is a function $\mathbb{T} \rightarrow \mathbb{R}^{d \times d}$,
$\boldsymbol{L}$ is a function $\mathbb{T} \rightarrow \mathbb{R}^{s \times d}$, and $H$ is a function $\mathbb{T} \rightarrow \mathbb{R}^{m \times d}$ where $\mathbb{T}$ is the "time dimension". In the following example, however, we consider a LTI SDE, namely, the Ornstein-Uhlenbeck Process from which we observe discrete linear Gaussian measurements.

### Define State-Space Model

#### I. Continuous Dynamics Model: Linear, Time-Invariant Stochastic Differential Equation (LTISDE)

```python
state_dim = 1
observation_dim = 1
```

```python
delta_t = 0.2
# Define Linear, time-invariant Stochastic Differential Equation that models
# the (scalar) Ornstein-Uhlenbeck Process
drift_constant = 0.21
dispersion_constant = np.sqrt(0.5)
drift = -drift_constant * np.eye(state_dim)
force = np.zeros(state_dim)
dispersion = dispersion_constant * np.eye(state_dim)
```

The _continuous_ counterpart to the discrete LTI Gaussian model from above is provided via the `LTISDE` class.
It is initialized by the state space components
- `drift_matrix` : the drift matrix $\boldsymbol{F}$
- `force_vector` : a force vector that is added to the state (note that this is **not** $\boldsymbol{\omega}$.) Here: zero.
- `dispersion_matrix` : the dispersion matrix $\boldsymbol{L}$

```python
# Create dynamics model
dynamics_model = randprocs.markov.continuous.LTISDE(
    drift_matrix=drift,
    force_vector=force,
    dispersion_matrix=dispersion,
)
```

#### II. Discrete Measurement Model: Linear, Time-Invariant Gaussian Measurements

```python
measurement_marginal_variance = 0.1
measurement_matrix = np.eye(observation_dim, state_dim)
measurement_noise_matrix = measurement_marginal_variance * np.eye(observation_dim)
```

As above, the measurement model is discrete, LTI Gaussian. Only the dymanics are continuous (i.e. continuous-discrete).

```python
noise = randvars.Normal(mean=np.zeros(observation_dim), cov=measurement_noise_matrix)
measurement_model = randprocs.markov.discrete.LTIGaussian(
    transition_matrix=measurement_matrix,
    noise=noise,
)
```

#### III. Initial State Random Variable

```python
mu_0 = 10.0 * np.ones(state_dim)
sigma_0 = np.eye(state_dim)
initial_state_rv = randvars.Normal(mean=mu_0, cov=sigma_0)
```

```python
prior_process = randprocs.markov.MarkovProcess(
    transition=dynamics_model, initrv=initial_state_rv, initarg=0.0
)
```

### Generate Data for the State-Space Model

`statespace.generate_artificial_measurements()` is used to sample both latent states and noisy observations from the specified state space model.

```python
time_grid = np.arange(0.0, 10.0, step=delta_t)
```

```python
latent_states, observations = randprocs.markov.utils.generate_artificial_measurements(
    rng=rng,
    prior_process=prior_process,
    measmod=measurement_model,
    times=time_grid,
)
```

```python
regression_problem = TimeSeriesRegressionProblem(
    observations=observations,
    locations=time_grid,
    measurement_models=[measurement_model] * len(time_grid),
)
```

### Kalman Filtering

In fact, since we still consider a **linear** model, we can apply Kalman Filtering in this case again.
According to Section 10 in [1], the moments of the filtering posterior in the continuous-discrete case are solutions to linear differential equations, which `probnum` solves for us when invoking the `<Kalman_object>.filtsmooth(...)` method.

#### I. Kalman Filter

```python
kalman_filter = filtsmooth.gaussian.Kalman(prior_process)
```

#### II. Perform Kalman Filtering + Rauch-Tung-Striebel Smoothing

```python
state_posterior, _ = kalman_filter.filtsmooth(regression_problem)
```

The method `filtsmooth` returns a `KalmanPosterior` object which provides convenience functions for e.g. sampling and prediction.
We can also extract the just computed posterior smoothing state variables by querying the `.state_rvs` property.
This yields a list of Gaussian Random Variables from which we can extract the statistics in order to visualize them.

```python
grid = np.linspace(0, 11, 500)

posterior_state_rvs = state_posterior(
    grid
)  # List of <num_time_points> Normal Random Variables
posterior_state_means = posterior_state_rvs.mean.squeeze()  # Shape: (num_time_points, )
posterior_state_covs = posterior_state_rvs.cov  # Shape: (num_time_points, )

samples = state_posterior.sample(rng=rng, size=3, t=grid)
```

### Visualize Results

```python
state_fig = plt.figure()

ax = state_fig.add_subplot()

# Plot means
ax.plot(grid, posterior_state_means, label="posterior mean")

# Plot samples
for smp in samples:
    ax.plot(
        grid,
        smp[:, 0],
        color="gray",
        alpha=0.75,
        linewidth=1,
        linestyle="dashed",
        label="sample",
    )

# Plot marginal standard deviations
std_x = np.sqrt(np.abs(posterior_state_covs)).squeeze()
ax.fill_between(
    grid,
    posterior_state_means - 1.96 * std_x,
    posterior_state_means + 1.96 * std_x,
    alpha=0.2,
    label="1.96 marginal stddev",
)
ax.scatter(time_grid, observations, marker=".", label="measurements")
# Add labels etc.
ax.set_xlabel("t")
ax.set_title(r"$x$")

# These two lines just remove duplicate labels (caused by the samples) from the legend
handles, labels = ax.get_legend_handles_labels()
by_label = dict(zip(labels, handles))

ax.legend(
    by_label.values(), by_label.keys(), loc="center left", bbox_to_anchor=(1, 0.5)
)

state_fig.tight_layout()
```

```python

```


# Optimización de ganancía
  + Buscar la manera más optima de obtener la ganancía deseada.
## Objetivo general

   + Lo que buscamos con este es proyecto es obtener 1,000 pesos mexicanos como ganancía después de comprar y vender monedas al tipo de cambio del día de hoy.
## Objetivos especificos

 ### Definición de proyecto
   + Definir las variables.
   + Definir las restricciones.
   + Definir la función a maximizar.
 ### Programado
   + Aplicar los aprendizajes obtenidos en el modulo 1, con respecto a la programación lineal y resolverlo programando con python.

## Las modenas y su tipo de cambio que tomaremos en cuenta son las siguientes:

```python
import sympy as sym #Importamos librerias
import numpy as np
from scipy.optimize import linprog
```

```python
f1 = np.array([[-1,-1,-1,-1,0.052,0,0,0,0.046,0,0,0,5.62,0,0,0,0.041,0,0,0]])
f2 = np.array([[19.1,0,0,0,-1,-1,-1,-1,0.87,0,0,0,107.32,0,0,0,0.79,0,0,0]])
f3 = np.array([[0,21.77,0,0,0,1.14,0,0,-1,-1,-1,-1,0,0,122.3,0,0,0,0.88,0]])
f4 = np.array([[0,0,0.18,0,0,0,0.0093,0,0,0,0.0083,0,-1,-1,-1,-1,0,0,0,0.0073]])
f5 = np.array([[0,0,0,24.38,0,0,0,1.28,0,0,0,1.12,0,0,0,136.74,-1,-1,-1,-1]])
b = np.array([[0,0,0,0,1000]])

```

```python
A=np.concatenate ((-f1,-f2),axis=0)
A=np.concatenate ((A,-f3),axis=0)
A=np.concatenate ((A,-f4),axis=0)
A=np.concatenate ((A,-f5),axis=0)

```

```python
resultado = opt.linprog(-f1[0],A_ub=A,b_ub=b)
resultado
```

```python

```


# Model comparison with PyMC3
### Brett Morris

EEG Meeting, 27 April 2021

# Setup
Install the necessary packages with:

```bash
conda install -c conda-forge pymc3
pip install pymc3-ext corner arviz
```

**Troubleshooting**: Getting PyMC3 up and running can be complicated, see these pages for the most up-to-date notes on how to install on: [Mac](https://github.com/pymc-devs/pymc3/wiki/Installation-Guide-(MacOS)), [Linux](https://github.com/pymc-devs/pymc3/wiki/Installation-Guide-(Linux)), [Windows](https://github.com/pymc-devs/pymc3/wiki/Installation-Guide-(Windows)). If those instructions fail, you can usually get quick help on this [PyMC discourse page](https://discourse.pymc.io).

# Background

Often in observational astronomy, we have several plausible physical models which could fit the data, and we wish to identify which model best fits the data with the fewest free parameters. This is often the case with CHEOPS observations, where the observations can often be modeled as a linear combination of several *basis vectors*, or vectors of measurements like the telescope temperature or time, with the same length as your time series measurements $y$. **See also** [this repo](https://github.com/bmorris3/photometry_linalg) which I presented at a previous group meeting on photometric reductions with linear regression.

### Linear algebra recap

In a linear observation model, we often suppose that our observations $y(t)$ can be approximated with a linear combination of the $N$ basis vectors $a_i(t)$ like so:

$$ y(t) = w_0 a_0(t) + w_1 a_1(t) + ... + w_N a_N(t)$$

where the $w_i$ coefficients are the *weights* on each basis vector which minimize the residuals between the observations and the linear combination of basis vectors. This is a least-squares problem, which can be phrased like so: the design matrix $X$,

\begin{equation}
{\bf X}  = \begin{bmatrix}
a_0(t_0) & a_1(t_0) & ... & a_N(t_0)\\
a_0(t_1) & a_1(t_1) & ... & a_N(t_1)\\
\vdots & \vdots & \vdots & \vdots \\
a_0(t_n) & a_1(t_n) & ... & a_N(t_n)\\
\end{bmatrix}
\end{equation}

is the concatenation of each basis column vector $a_i$. We want to solve for the weights
\begin{equation}
{\bf \hat{\beta}} = \left[ w_0 ~ w_1 ~ ... ~  w_N \right]
\end{equation}
which minimize

$$ \arg \min_\beta || {\bf y - X \hat{\beta}} ||. $$

The best-fit estimators $\hat{\beta}$ can be solved with some matrix math (see [Wikipedia](https://en.wikipedia.org/wiki/Ordinary_least_squares) for more):

$${\displaystyle {\hat {\boldsymbol {\beta }}}=(\mathbf {X} ^{\rm {T}} N^{-1} \mathbf {X} )^{-1}\mathbf {X} ^{\rm {T}} N^{-1} \mathbf {y}, }$$

where $\bf N = \sigma_y^2 {\bf I}$ is the observational uncertainty on each measurement $\sigma_y$, multiplied by the identity matrix $\bf I$.

### Why use PyMC3?

> PyMC3 is a new, open-source [pure Python] framework with an intuitive and readable, yet powerful, syntax that is close to the natural syntax statisticians use to describe models. It features next-generation Markov chain Monte Carlo (MCMC) sampling algorithms such as the **No-U-Turn Sampler** (NUTS; Hoffman, 2014), a self-tuning variant of Hamiltonian Monte Carlo (HMC; Duane, 1987). This class of samplers works well on **high dimensional and complex posterior distributions** and allows many complex models to be fit without specialized knowledge about fitting algorithms.

> While most of PyMC3’s user-facing features are written in pure Python, it leverages Theano (Bergstra et al., 2010) to transparently **transcode models to C** and compile them to machine code, thereby boosting performance.

Source: [PyMC3 docs](https://docs.pymc.io/notebooks/getting_started.html), see also the [Example Gallery](https://docs.pymc.io/nb_examples/index.html), and [this manuscript on HMC methods](https://arxiv.org/abs/1701.02434)

# Example with [PyMC3](https://docs.pymc.io)
Next let's import the necessary packages:

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pymc3 as pm
import pymc3_ext as pmx
import arviz as az
from corner import corner
```

Generate an example dataset with three components: linear, quadratic, and sinusoidal trends. Add random noise:

```python
np.random.seed(42)

n = 1000
x = np.linspace(-6, 6, n)

truths = [
    2,    # DC offset of the whole curve
    3,    # linear trend weight
    0.2,  # quadratic trend weight
    2     # sinusoidal weight
]

true_linear = truths[0] + truths[1] * x
true_period = 0.8
true_quadratic = truths[2] * x**2
true_sinusoid = truths[3] * np.sin(2 * np.pi * x / true_period)

uncertainties = 0.5
noise = np.random.normal(scale=0.5, size=n)

y = (
    true_linear +
    true_quadratic +
    true_sinusoid
) + noise

plt.figure(figsize=(8, 4))
plt.plot(x, y, 'k.', label='observations')
plt.plot(x, true_linear, label='linear')
plt.plot(x, true_sinusoid, label='sin')
plt.plot(x, true_quadratic, label='quad')
plt.legend()
plt.gca().set(xlabel='$x$', ylabel='$y$')
```

Create a list of (unscaled) basis vectors, which we will use to solve for the linear regression coefficients (weights):

```python
basis_vectors = [
    np.ones_like(x), # DC offset
    x,               # linear trend
    x**2,            # quadratic trend
    np.sin(2 * np.pi * x / true_period), # sinusoid
     -(np.abs(x) < 2).astype(int) # extra, unnecessary vector
]

basis_vector_names = [
    "unit",
    "$x$",
    "$x^2$",
    "$\sin$",
    "box"
]
```

Loop through basis vectors, adding one at a time, and evaluating the best-fit linear regression coefficients for each combination of basis vectors. Save results to a dictionary called ``inference_data_dict``, and plot the *maximum a posteriori* (MAP) solution in red over the data in black:

```python
total_basis_vectors = len(basis_vectors)

inference_data_dict = {}

for n_basis_vectors in range(1, total_basis_vectors + 1):

    model_name = f"n={n_basis_vectors}"

    with pm.Model(name=model_name) as pymc3_model:

        beta = pm.Uniform("beta", lower=-10, upper=10, shape=n_basis_vectors)

        linear_model = pm.math.dot(beta, basis_vectors[:n_basis_vectors])

        pm.Normal("obs", mu=linear_model, sigma=uncertainties, observed=y)

        map_soln = pm.find_MAP()

        plt.plot(x, y, '.k')
        plt.plot(x, pmx.eval_in_model(linear_model, map_soln), 'r')
        plt.title("Basis vectors: " + ", ".join(basis_vector_names[:n_basis_vectors]))
        plt.gca().set(xlabel='$x$', ylabel='$y$')
        plt.show()

        inference_data = pm.sample(
            draws=1000,
            tune=1000,
            compute_convergence_checks=False,
            return_inferencedata=True,
            target_accept=0.9,
            cores=2
        )

        inference_data_dict[model_name] = inference_data
```

## Model comparison

Now that we've solved for the best-fit weights on each basis vector for all five basis vectors. Which combination of basis vectors best reproduces the observations? **Which model do we choose?**

One technique for measuring the relative confidence that a set of models fits the data is the [Leave-One-Out cross validation statistic](https://en.wikipedia.org/wiki/Cross-validation_(statistics)#Leave-one-out_cross-validation). In its simplest form, the LOO will do the following:

1. Divide the observations into test and training groups of length $1$ and $n-1$
2. Fit the model on the $n-1$ training observations, predict measurement at the "missing" data point
3. Compare the prediction at the "missing" point to the actual observation
4. Sum the squares of these residuals and normalize by the number of data points

```python
comparison_loo = az.compare(
    inference_data_dict,
    ic='loo'
)

comparison_loo
```

For illustrative purposes, we compute the leave-one-out cross validation statistic in the [style of the wikipedia entry](https://en.wikipedia.org/wiki/Cross-validation_(statistics)#Leave-one-out_cross-validation) for each model explicitly here:

```python
# Loop over the number of basis vectors:
for n_basis_vectors in range(1, len(basis_vectors) + 1):

    # Construct a design matrix out of the first N basis vectors
    design_matrix = np.vstack(basis_vectors[:n_basis_vectors]).T

    # Compute the LOO by leaving out one data point, refitting the
    # model, and subtracting the best-fit expectation from the
    # actual observation at the missing data point:
    loo_statistic = 0
    for i in range(len(x)):
        indices = np.r_[0:i-1, i+1:len(x)]   # indexing trick to skip `i`th index
        x_in = x[indices]
        y_in = y[indices]
        design_matrix_in = design_matrix[indices, :]

        # Compute the least squares regression:
        betas = np.linalg.lstsq(design_matrix_in, y_in, rcond=-1)[0]

        # Compute the value of the model at the missing time
        y_out = design_matrix[i, :] @ betas

        # Compute the difference between expected and observed at
        # the missing data point:
        loo_statistic += (y[i] - y_out)**2

    # Normalize over the length of the dataset:
    loo_statistic /= len(x)
    print(f"LOO for n={n_basis_vectors}: {loo_statistic}")
```

Note that the LOO is smallest for the $n=\{4,5\}$ models, and in that case, you may want to select the model with the fewest free parameters, in this case $n=4$.

We can also compute other information criteria, such as the "widely applicable information criterion" (read [all of the details here](http://www.stat.columbia.edu/~gelman/research/published/waic_understand3.pdf)) with arviz:

```python
comparison_waic = az.compare(
    inference_data_dict,
    ic='waic'
)

comparison_waic
```

If you compare the WAIC stats to the LOO stats above, you'll see that they agree closely.

Finally, let's make a corner plot of the $n=4$ model to see how well the best-fit predictions for ${\bf \hat{\beta}}$ compare with the true (input) values:

```python
# Convert the posteriors to a flat chain for plotting a corner plot:
post = inference_data_dict['n=4'].posterior['n=4_beta'].values
flatchain = post.reshape((post.shape[0] * post.shape[1], -1))
```

```python
corner(flatchain, labels=basis_vector_names, truths=truths);
```

The agreement is quite good!

```python

```


# Занятие 7
# Прикладная алгебра и численные методы
## Многочлены Чебышева

```
!python -m pip install sympy --upgrade
```

```
!python -m pip install -U matplotlib
```

```
import numpy as np
import sympy
from sympy import S
from sympy.functions.special.polynomials import chebyshevt, chebyshevu
from numpy.polynomial.chebyshev import chebinterpolate
import matplotlib.pyplot as plt
import pandas as pd
from google.colab import files
%matplotlib inline
```

```
import sympy
import numpy
sympy.__version__, numpy.__version__
```

https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.norm.html#scipy.linalg.norm

https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html

## Многочлен Чебышева первого рода
$T_0(x) = 1$, $T_1(x) = x$, $T_(x) = 2xT_{n - 1}(x) - T_{n - 2}(x)$, $n\ge 2$.
## Задание 1
Построить графики многочленов Чебышева первого рода при $n = 1 - 5$, каждый график отдельным цветом, в легенду включить формулу многочлена в форме $T_1(x) = x$.

!!! Использовать matplotlib, lambdify, Eq и многочлены Чебышева chebyshevt из Sympy !!!

```

```

## Задание 2.
$$
T_n(x) = \frac{(x + \sqrt{x^2 - 1})^n + (x - \sqrt{x^2 - 1})^n}{2}, \quad |x| \ge 1.
$$
Построить многочлен Чебышева первого рода порядка 9, 11 и 13 по приведенной выше формуле и с помощью chebyshevt из Sympy.

Для каждого порядка 9, 11 и 13 на сетке значений $x$ от 5 до 6 с шагом 0.001 вычислить нормы 1 и 2 разностей значений многочлена Чебышева, полученного по формуле и с помощью chebyshevt.

Изобразить на отдельных графиках многочлены каждого порядка, полученные двумя способами (получится 3 графика)

Использовать plt.show() для изображения отдельного графика и plt.close для того, чтобы следующий график не содержал только что выведенного рисунка.

```

```

## Задание 3
Многочлен Чебышева второго рода:
$$
U_n = \frac{1}{n + 1}T'_{n + 1}(x), \quad n \ge 0.
$$
$$
U_n(x) = \frac{(x + \sqrt{x^2 - 1})^{n + 1} - (x - \sqrt{x^2 - 1})^{n + 1}}{2\sqrt{x^2 - 1}}, \quad |x| \ge 1.
$$

Построить многочлен Чебышева степени 10 по приведенным двум формулам, за основу в первой формуле взять chebyshevt из sympy.
Построить многочлен Чебышева степени 10 с помощью chebyshevu sympy.

При необходимости упростить полученные выражения и сравнить их.

```

```

## Задание 4
Норма Чебышева (максимум модуля на данном отрезке):
$$
|f|_0 = \max_{[-1, 1]}|f(x)|.
$$
Площадь под графиком функции на данном отрезке:
$$
|f|_1 = \int_{-1}^1|f(x)|\,dx.
$$
Вычислить норму Чебышева и площадь под графиком для $\sin(\pi x)$ на отрезке [-1, 1].

```

```

## Задание 5
Наилучшее приближение функции $f(x)$ многочленом степени $\le n$:
$$
\tilde{f}(x) = \sum_{i=0}^n \frac{<T_i, f>}{<T_i, T_i>}T_i(x),
$$
$T_i(x)$ многочлен Чебышева первого рода степени $i$,
$$
<f, g> = \int_{-1}^1 \frac{f(x)g(x)}{\sqrt{1 - x^2}}\,dx, \quad
|f| = \sqrt{<f, f>} = \sqrt{\int_{-1}^1 \frac{f^2(x)}{\sqrt{1 - x^2}}\,dx}
$$

Построить с помощью Sympy и Numpy многочлен степени не выше 4, наилучшим образом приближающий $x^6 - x^2 + 2$ на [-1, 1].

Изобразить на общем графике $x^6 - x^2 + 2$ и приближения многочленами Чебышева, в легенду включить формулы.

Вычислить норму Чебышева и площадь между графиками функции и ее представления полиномами Чебышева.

```

```


```python
#Import and set magics
import numpy as np
%matplotlib inline
import pandas as pd
import datetime
import pandas_datareader
import pydst
import statsmodels.api as stat
import statsmodels.formula.api as statf
from statsmodels.sandbox.regression.predstd import wls_prediction_std
import sympy as sm
import math
import random

import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
from matplotlib import cm
%matplotlib inline
from mpl_toolkits.mplot3d import Axes3D

```

# Estimating income processes

Consider $N$ households indexed by $i$ who are in the labor market for $T$ periods indexed by $t \in ${1,2,...,T}$,

Their **wage income** is stochastic and given by,

$P_{i,t}$ = $\psi_{i,t}$ $P_{i,t-1}$

$\tilde{Y}_{i,t}$ = $xi_{i,t}$ $P_{i,t}$

$Y_{i,t}$ = \left\begin{array}{rcl}$
            \overline{\overline{0}+\text{if}\mu_{i,t}<\pi}
            \\tilde{Y}_{i,t} +\text{else}
            \end{array}\right.

$\psi_{i,t}$ ~ LogNormal(-0.5$\sigma_{\psi}^2$,$\sigma_{\psi})$

$\xi_{i,t}$ ~ LogNormal(-0.5$\sigma_{\xi}^2$,$\sigma_{\xi})$

$\mu_{i,t}$ ~ $Uniform(0,1)$

$P_{0} = 1$

where

$\sigma_{\psi}$ is the standard deviation of the *permanent* shocks, $\psi_{i,t}$

$\sigma_{\xi}$ is the standard deviation of the *transitory* shocks, $\xi_{i,t}$

$\pi$ is the risk of unemployment

The data you have access to is:

```python
import numpy as np
dataY = np.load(r'C:\Users\musse\Documents\GitHub\projects-2020-zjr-hrv\projects-2020-zjr-hrv\dataY.npy')
T,N = dataY.shape
print(dataY)
print(dataY.shape)

```

    [[0.83317569 0.72881172 0.         ... 1.16771806 0.93207083 0.86711803]
     [1.18542294 0.92826337 1.62913142 ... 1.13903869 0.94479246 0.78842682]
     [1.14813596 0.90542496 0.70634401 ... 1.49584791 1.08969956 0.        ]
     ...
     [0.73818651 0.59958039 0.56135238 ... 2.60075173 1.07070489 0.43010036]
     [1.14130879 0.85728537 0.54530761 ... 3.79294856 0.67764143 0.38720822]
     [0.64908127 0.85101393 0.59517395 ... 3.32800991 0.82400879 0.5814573 ]]
    (20, 50000)

**Question 1:** Calculate income growth rates as log-changes

```python
#Defining income growth rate by finding the first differences. Thereafter replacing the negative Y-values with .nan

logY = np.diff(np.log(dataY))
#diff_dataY = np.diff(dataY)

# printing initial arrays
print("Initial diff", logY)

# code to replace all negative value with 0
logY[logY<=0] = np.nan

# printing result
print("New resulting diff: ", logY)

```

    Initial diff [[-0.13382911        -inf         inf ...  0.45991757 -0.22539794
      -0.0722337 ]
     [-0.2445394   0.56248678 -0.45247293 ...  0.10650339 -0.18697465
      -0.18092569]
     [-0.2374906  -0.24830202  0.49367077 ...  0.38133059 -0.31679118
             -inf]
     ...
     [-0.20796646 -0.06588123  0.6550227  ...  1.08826931 -0.88748332
      -0.9120539 ]
     [-0.2861601  -0.45242079  0.43603115 ...  1.35354726 -1.7222807
      -0.5596557 ]
     [ 0.27087056 -0.35757477  0.31896575 ...  1.52939372 -1.39594858
      -0.34864367]]
    New resulting diff:  [[       nan        nan        inf ... 0.45991757        nan        nan]
     [       nan 0.56248678        nan ... 0.10650339        nan        nan]
     [       nan        nan 0.49367077 ... 0.38133059        nan        nan]
     ...
     [       nan        nan 0.6550227  ... 1.08826931        nan        nan]
     [       nan        nan 0.43603115 ... 1.35354726        nan        nan]
     [0.27087056        nan 0.31896575 ... 1.52939372        nan        nan]]

**Question 2:** Calculate the following 3 statistics from the data:

$s_{1}$: Share of observations with $Y_{i,t}$=0

$s_{2}$: Variance of income growth rate, Var($\Delta$ Log $Y_{i,t}$)

$s_{3}$: Covariance of income growth rates one period apart, Cov($\Delta$ Log $Y_{i,t}$,$\Delta$ Log $Y_{i,t-1}$ )

```python
#First, we define moments to use in both question 2 and 3:
def momments(Y):
    s_sim1 = np.count_nonzero(Y==0)/np.size(Y)
    Y[Y<0] = np.nan
    Ydiff = np.diff(np.log(Y))
    cov = pd.DataFrame(Ydiff).cov()

    s_sim2 = np.zeros(Ydiff.shape[1])
    for i in range(Ydiff.shape[1]):
        s_sim2[i] = cov.iloc[i,i]
    s_sim2 = np.mean(s_sim2)

    s_sim3 = np.zeros(Ydiff.shape[1]-1)
    for i in range(Ydiff.shape[1]-1):
        s_sim3[i] = cov.iloc[i,(i+1)]
    s_sim3 = np.mean(s_sim3)

    return s_sim1, s_sim2, s_sim3
```

The figures in question 2 are now calculated as follows:

```python
#S1_data
s_data = [None]*3
#S2_data
s_data[0], s_data[1], s_data[2] = momments(dataY.T)
#S3_data
s_data
```

    [0.049925, 0.054993983413371354, -0.022504802437365256]

Question 3: Simulate the income process using your own choice of $\sigma_{\psi}$, $\sigma_{\xi}$, $\pi$, T and N. Calculate the 3 same statistics. Compare with the data statistics

```python

#Simulated figures
T = 50
N = 1000
pi = 0.3
s_psi = 0.23
s_xi = 0.4
P_0 = 1

#Defining simulation
def sim(s_psi,s_xi,pi,N,T):
    #Model
    Y = np.zeros((N,T))
    np.random.seed(1)
    for t in range(T):
        psi = np.random.lognormal(-0.5 * s_psi, s_psi, N)
        xi = np.random.lognormal(-0.5 * s_xi,s_xi, N)
        P = psi * P_0
        Y_tilde = xi * P
        mu = np.random.uniform(0,1,N)
        Y[(pi<mu),t] = Y_tilde[(pi<mu)]
    return Y

Y_sim = sim(s_psi,s_xi,pi,N,T)
#S1_sim
s_sim = [None]*3
#S2_sim
s_sim[0], s_sim[1], s_sim[2] = momments(Y_sim)
#S3_sim
s_sim

```

    [0.29988, 0.425019139369132, -0.21104509754699205]

Comparison of the simulated data and the factual dataset

s1_sim =0.29988

s1_data = 0.049925

s2_sim = 0.425019139369132

s2_data = 0.054993983413371354

s3_sim = -0.21104509754699205

s3_data = -0.022504802437365256

Overall, the dataset has lower numbers than the simulated data.

**Question 4:** Solve the following minimization problem to estimate $\sigma_{\psi}$, $\sigma_{\xi}$ and $\pi$

$\sigma_{\psi}^*$, $\sigma_{\xi}^*$, $\pi^*$ = $arg_{\sigma_{\psi}>=0,\sigma_{\xi}>=0,\pi\in [0,1]}$               min $(s_{1}^{sim}-s_{1}^{data})^2$ + $(s_{2}^{sim}-s_{2}^{data})^2$ + $(s_{3}^{sim}-s_{3}^{data})^2$

where for each new guess of $\sigma_{\psi}$, $\sigma_{\xi}$ and $\pi$ you should be re-simulating the data with the same seed and re-calculate the 3 statistics

```python
#Defining the objective function
def objective(s):

    Y = sim(s[0],s[1],s[2],N,T)
    s_sim1, s_sim2, s_sim3 = momments(Y)
    #obj =  (s_sim1 - s_data1)**2 + (s_sim2 - s_data2)**2 + (s_sim3 - s_data3)**2
    obj =   (s_sim1 - s_data[0])**2 + (s_sim2 - s_data[1])**2 + (s_sim3 - s_data[2])**2
    return obj

x0 = np.ones(3)/100

#Minimizing
res = optimize.minimize(objective,x0,method='L-BFGS-B',options={'eps':1e-4},bounds=((0,None),(0,None),(0,None),))
res
```

          fun: 1.6144875720339336e-05
     hess_inv: <3x3 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 2.25305304e-05,  6.20294454e-06, -2.49368487e-04])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 104
          nit: 16
       status: 0
      success: True
            x: array([0.11493023, 0.11775912, 0.04956846])

# Wealth in the utility function

In the final period, t=T, the household solves the following problem:

$$
\begin{aligned}
v_{T}(a_{T-1})&= \max_{c_{T}}\frac{(c_{T}^{1-\rho})}{1-\rho} + \kappa \frac{(a_{T}+a\underline)^{1-\sigma}}{1-\sigma} \\
\text{s.t.} \\
a_{T} = (1+r)a_{T-1}+y-c_{T}
\end{aligned}
$$

where

 * $a_t$ is end-of-period assets in period $t$
 * $c_t$ is consumption in period $t$
 * $\rho$ is the CRRA-coefficient for consumption utility
 * $\sigma$ is the CRRA-coefficient for wealth utility
 * $\underline{a}$ is an *additive* scaling factor for wealth utility
 * $\kappa$ is a *multiplicative* scaling factor for wealth utility
 * $r$ is the rate of return
 * $y$ is income

The optimal **consumption function** is denoted $c_t^{*}(a_{t-1})$

The optimal **savings function** is denoted $a_t^{*}(a_{t-1}) = (1+r)a_{t-1} + y - c_t^{*}(a_{t-1})$.

```python
# a. parameters
rho = 2.0
sigma = 1.2
kappa = 0.6
a_ubar = 2.0
r = 0.04
y = 1.0
# b. grids
a_lag_vec = np.linspace(0,300,300)

```

**Question 1:** Find and plot the functions $v_{T}(a_{T-1})$, $c_T^{*}(a_{T-1})$, and $a_T^{*}(a_{T-1})$

```python
#Definining the given utility function
def utility (c,y,r,rho,kappa,a_lag,a_ubar,sigma):
    return c**(1-rho)/(1-rho) + (kappa*(((1+r)*a_lag+y-c)+a_ubar)**(1-sigma))/(1-sigma)

```

```python
#Defining the solution function for period T
def solve_period_T(rho,kappa,a_ubar,sigma,y):

    vT_grid = np.empty(a_lag_vec.size)
    cT_grid = np.empty(a_lag_vec.size)
    aT_grid = np.empty(a_lag_vec.size)

    # solve for a_lag in grid
    for i, a_lag in enumerate(a_lag_vec):

        # Objective function
        obj = lambda c: -utility(c,y,r,rho,kappa,a_lag_vec[i],a_ubar,sigma)

        #Initial guess of consumption
        x0 = a_lag_vec[i]/10 + y
        #Optimization
        result = optimize.minimize(obj,[x0],method='L-BFGS-B')
        #Saving our results
        vT_grid[i] = -result.fun
        cT_grid[i] = result.x
        aT_grid[i] = (1+r)*a_lag_vec[i]+y-cT_grid[i]

    return vT_grid,cT_grid,aT_grid

```

```python
v_grid =  [None]*(T+1)
c_grid = [None]*(T+1)
a_grid = [None]*(T+1)
v_interp_list =  [None]*(T+1)
```

```python
#Solving the max problem in period T
v_grid[T],c_grid[T],a_grid[T] = solve_period_T(rho,kappa,a_ubar,sigma,y)

```

```python
#Constructing interpolator
v_interp_list[T] = interpolate.RegularGridInterpolator([a_lag_vec],v_grid[T], bounds_error=False)
```

```python
#Plotting vT

fig, ax = plt.subplots()
ax.plot(a_lag_vec,v_grid[T])

ax.set(xlabel='$[a_T]$', ylabel='$v_T[aT-1]$')
ax.grid()

fig.savefig("$v_T$")
plt.show()

```

```python
#Plotting cT

fig, ax = plt.subplots()
ax.plot(a_lag_vec,c_grid[T])

ax.set(xlabel='$[a_T]$', ylabel='$c^*_T[aT-1]$')
ax.grid()

fig.savefig("$c_T$")
plt.show()

```

```python
#Plotting aT

fig, ax = plt.subplots()
ax.plot(a_lag_vec,a_grid[T])

ax.set(xlabel='$a_[T-1]$', ylabel='$a_T^*(aT-1)$')
ax.grid()

fig.savefig("$a_T$")
plt.show()

```

**Question 2:** Find and plot $v_{T-1}(a_{T-2})$ and  $c_{T-1}^{*}(a_{T-2})$.

In all periods *before the last*, $t < T$, the household solves:

$v_{t}(a_{t-1})$ = $max_{c_{t}} \frac{c_t^{1-\rho}}{1-\rho} + \kappa \frac{(a_t+\underline{a})^{1-\sigma}}{1-\sigma} + \beta v_{t+1}(a_t)$

{s.t.}
    $a_{t} = (1+r)a_{t-1} + y - c_{t}$

where $\beta$ is the discount factor for future utility.

```python
beta = 0.97
T= 20
```

```python
#solve function for period t
def solve_period_t(rho,kappa,a_ubar,sigma,y,t):

    vt_grid = np.empty(a_lag_vec.size)
    ct_grid = np.empty(a_lag_vec.size)
    at_grid = np.empty(a_lag_vec.size)

    #solve for each lag in grid
    for i, a_lag in enumerate(a_lag_vec):

        #objective function
        obj = lambda c: -utility(c,y,r,rho,kappa,a_lag_vec[i],a_ubar,sigma) - beta*v_interp_list[t+1]([((1+r)*a_lag+y-c)])
        #Initial guess
        x0 = a_lag_vec[i]/10 + y
        #optimazation
        result = optimize.minimize(obj,[x0],method='L-BFGS-B',bounds=((1e-8,None),))
        #saving results
        vt_grid[i] = -result.fun
        ct_grid[i] = result.x
        at_grid[i] = (1+r)*a_lag_vec[i]+y-ct_grid[i]

    return vt_grid,ct_grid,at_grid
```

```python
#Creating a loop to calculate for both question 2 and 3
for i in range(T):

    t = T-i-1
    v_gridt,c_gridt,a_gridt = solve_period_t(rho,kappa,a_ubar,sigma,y,t)
    v_grid[t] = v_gridt
    c_grid[t] = c_gridt
    a_grid[t] = a_gridt
    v_interp_list[t] = interpolate.RegularGridInterpolator([a_lag_vec],v_grid[t], bounds_error=False, fill_value=None)

```

```python
#Plotting vt
fig, ax = plt.subplots()
ax.plot(a_lag_vec,v_grid[19])

ax.set(xlabel='$a_[T-2]$', ylabel='$v_T^*(aT-2)$')
ax.grid()

fig.savefig("$v_[T-1]$")
plt.show()
```

```python
#Plotting c_t-1

fig, ax = plt.subplots()
ax.plot(a_lag_vec,c_grid[19])

ax.set(xlabel='$a_[T-2]$', ylabel='$c_T^*(aT-2)$')
ax.grid()

fig.savefig("$c_[T-1]$")
plt.show()
```

**Question 3:** Find $c_t^{*}(a_{t-1})$ for $t \in {0,1,...,T}$ and plot them in a single figure.

```python
#for i in range(T):

    #t = T-i-1
    #v_gridt,c_gridt,a_gridt = solve_period_t(rho,kappa,a_ubar,sigma,y,t)
    #v_grid[t] = v_gridt
    #c_grid[t] = c_gridt
    #a_grid[t] = a_gridt
    #v_interp_list[t] = interpolate.RegularGridInterpolator([a_lag_vec],v_grid[t], bounds_error=False, fill_value=None)

```

```python
#Plotting c_t from 1 to T
plt.plot (a_lag_vec,c_grid[1])
plt.plot (a_lag_vec,c_grid[2])
plt.plot (a_lag_vec,c_grid[3])
plt.plot (a_lag_vec,c_grid[4])
plt.plot (a_lag_vec,c_grid[5])
plt.plot (a_lag_vec,c_grid[6])
plt.plot (a_lag_vec,c_grid[7])
plt.plot (a_lag_vec,c_grid[8])
plt.plot (a_lag_vec,c_grid[9])
plt.plot (a_lag_vec,c_grid[10])
plt.plot (a_lag_vec,c_grid[11])
plt.plot (a_lag_vec,c_grid[12])
plt.plot (a_lag_vec,c_grid[13])
plt.plot (a_lag_vec,c_grid[14])
plt.plot (a_lag_vec,c_grid[15])
plt.plot (a_lag_vec,c_grid[16])
plt.plot (a_lag_vec,c_grid[17])
plt.plot (a_lag_vec,c_grid[18])
plt.plot (a_lag_vec,c_grid[19])
plt.plot (a_lag_vec,c_grid[20])

plt.legend()

```

Define the saving rate as:

$s_{t}^* (a_{t-1})$ = $\frac{(a_{t}-a_{t-1}+y-c_{t}^*(a_{t-1}))-a_{t-1}}{y+ra_{t-1}}$

**Question 4:** Plot $s_0^{*}(a_{-1})$. Do the rich or the poor save the most?

```python
s_0 = (r*a_lag_vec + y - c_grid[0])/(y+r*a_lag_vec)
```

```python
#Plot savingsrate
fig = plt.figure(figsize=(10,4))
ax = fig.add_subplot(1,2,1)
ax.plot(a_lag_vec,s_0)
ax.set_xlabel('$a_(t-1)$')
ax.set_ylabel('$s_0$')
ax.set_title('Savingsrate')
```

The saving rate have a u shape. If you are really poor then you save a lot and when you very rich then you save a lot too. Yet there is a interval where not either very rich or poor, save that much. So it is difficult to say because of the shape of the savingsrate.

**Question 5:** Can you change the parameter choices such that $s_0^{*}(a_{-1})$ is monotonically decreasing in $a_{-1}$?

```python
#Changing the parameter r, to see if the saving rate is decreasing
r1=0
s_00 = (r1*a_lag_vec + y - c_grid[0])/(y+r1*a_lag_vec)
```

```python
fig = plt.figure(figsize=(10,4))
ax = fig.add_subplot(1,2,1)
ax.plot(a_lag_vec,s_00)
ax.set_xlabel('$a_(t-1)$')
ax.set_ylabel('$s_0$')
ax.set_title('Savingsrate')
```

By setting the rate of return equal to 0, then we get a monotonically decreasing in a_t-1

# Refined grid search

Let $\boldsymbol{x} = \left[\begin{array}{c}
x_1 \\
x_2\\
\end{array}\right]$ be a two-dimensional vector. Consider the following algorithm:

**Algorithm:** `grid_search()`

**Goal:** Minimize the function $f(\boldsymbol{x})$.

1. Choose a grid size $N$ and minimum and maximum values of $x_{1}$ and $x_{2}$ denoted $\overline{x_{1}}$>$\underline{x_{1}}$ and $\overline{x_{2}}$>$\underline{x_{2}}$
2. Calculate step sizes

    $\Delta_{1}$ =($\overline{x_{1}}$-$\underline{x_{1}}$)/(N-1)

    $\Delta_{2}$ =($\overline{x_{2}}$-$\underline{x_{2}}$)/(N-1)
3. Find the grid point with the lowest function value solving

    $j^{*}_{1}$, $j^{*}_{2}$ = $arg_{j_{1}\in{0,...,N-1}, j_{2}\in{0,...,N-1}}$  f( $\underline{x_{1}+j_{1}$\Delta_{1}$,$\underline{x_{2}+j_{2}$\Delta_{2}$)

1. Return

**Question 1:** Implement the grid_search() algorithm to minimize the rosen function.

```python
def rosen(x):
 return (1.0-x[0])**2+2*(x[1]-x[0]**2)**2

```

```python
# settings
x1_min = 0
x1_max = 5
x2_min = 0
x2_max = 4
N = 1000
#Implementing grid_search()
def grid_search(f,x1_min,x1_max,x2_min,x2_max,N):

    #calculate step sizes
    delta1=(x1_max-x1_min)/(N-1)
    delta2=(x2_max-x2_min)/(N-1)

    f_min = np.inf
    arg_x_min = None
    x = None

    #making a loop
    for i in range(N):
        for j in range(N):
            x = [x1_min+i*delta1,x2_min+j*delta2]
            f_val = f(x)
            if f_val < f_min:
                f_min = f_val
                arg_x_min = x

    return arg_x_min,f_min

# apply grid search
x,f = grid_search(rosen,x1_min,x1_max,x2_min,x2_max,N)
print('minimum found at' , x)
print('with the function value' ,f)
```

    minimum found at [1.001001001001001, 1.001001001001001]
    with the function value 3.0100230440752395e-06

**Question 2:** Implement the refined_grid_search() algorithm to minimize the rosen function

```python
K = 10
def refined_grid_search(f,x1_min,x1_max,x2_min,x2_max,N,K):
    x = [None]*2
    k = 0
    #making a loop
    for i in range(K):
        if k > 0:
            x1_delta = 3*(x1_max-x1_min)/(N-1)
            x2_delta = 3*(x2_max-x2_min)/(N-1)
            x1_min = np.maximum(x1_min,x[0]-x1_delta)
            x2_min = np.maximum(x2_min,x[1]-x2_delta)

            x1_max = np.minimum(x1_max,x[0]+x1_delta)
            x2_max = np.minimum(x2_max,x[1]+x2_delta)

        x,f_max = grid_search(f,x1_min,x1_max,x2_min,x2_max,N)
        k = k + 1
        if k >= K:

            return x, f_max

# apply refined grid search
x,f_max = refined_grid_search(rosen,x1_min,x1_max,x2_min,x2_max,N,K)
```

```python
np.minimum(1,2)
```

    1

```python
print('minimum found at', x)
print('at the function value', f_max)
```

    minimum found at [1.0, 1.0]
    at the function value 0.0


# Hartree-Fock Self-Consistent Field Theory

```julia
"""Tutorial implementing a basic Hartree--Fock SCF program."""

__authors__ = "D. Menendez", "D. A. Sirianni"
__credits__ = "D. Menendez", "D. G. A. Smith"
__email__   = "danielmail7@gmail.com"

__copyright__ = "(c) 2014-2020, The Psi4Julia Developers"
__license__   = "BSD-3-Clause"
__date__      = "07/28/2020"
```

    "07/28/2020"

## I. Theoretical Overview
In this tutorial, we will seek to introduce the theory and implementation of the quantum chemical method known as Hartree-Fock Self-Consistent Field Theory (HF-SCF) with restricted orbitals and closed-shell systems (RHF).  This theory seeks to solve the pseudo-eigenvalue matrix equation

$$\sum_{\nu} F_{\mu\nu}C_{\nu i} = \epsilon_i\sum_{\nu}S_{\mu\nu}C_{\nu i}$$
$${\bf FC} = {\bf SC\epsilon},$$

called the Roothan equations, which can be solved self-consistently for the orbital coefficient matrix **C** and the orbital energy eigenvalues $\epsilon_i$.  The Fock matrix, **F**, has elements $F_{\mu\nu}$ given (in the atomic orbital basis) as

$$F_{\mu\nu} = H_{\mu\nu} + 2(\mu\,\nu\left|\,\lambda\,\sigma)D_{\lambda\sigma} - (\mu\,\lambda\,\right|\nu\,\sigma)D_{\lambda\sigma},$$

where $D_{\lambda\sigma}$ is an element of the one-particle density matrix **D**, constructed from the orbital coefficient matrix **C**:

$$D_{\lambda\sigma} = C_{\sigma i}C_{\lambda i}$$

Formally, the orbital coefficient matrix **C** is a $N\times M$ matrix, where $N$ is the number of atomic basis functions, and $M$ is the total number of molecular orbitals.  Physically, this matrix describes the contribution of every atomic basis function (columns) to a particular molecular orbital (e.g., the $i^{\rm th}$ row).  The density matrix **D** is a square matrix describing the electron density contained in each orbital.  In the molecular orbital basis, the density matrix has elements

$$D_{pq} = \left\{
\begin{array}{ll}
2\delta_{pq} & p\; {\rm occupied} \\
0 & p\; {\rm virtual} \\
\end{array}\right .$$

The total RHF energy is given by

$$E^{\rm RHF}_{\rm total} = E^{\rm RHF}_{\rm elec} + E^{\rm BO}_{\rm nuc},$$

where $E^{\rm RHF}_{\rm elec}$ is the final electronic RHF energy, and $E^{\rm BO}_{\rm nuc}$ is the total nuclear repulsion energy within the Born-Oppenheimer approximation.  To compute the electronic energy, we may use the density matrix in the AO basis:

$$E^{\rm RHF}_{\rm elec} = (F_{\mu\nu} + H_{\mu\nu})D_{\mu\nu},$$

and the nuclear repulsion energy is simply

$$E^{\rm BO}_{\rm nuc} = \sum_{A>B}\frac{Z_AZ_B}{r_{AB}}$$

where $Z_A$ is the nuclear charge of atom $A$, and the sum runs over all unique nuclear pairs.

## II. Implementation

Using the above overview, let's write a RHF program using  Psi4 , NumPy, TensorOperations, and Julia's LinearAlgebra standard library.  First, we need to import these modules:

```julia
# ==> Import Psi4 & NumPy <==
using PyCall: pyimport
psi4 = pyimport("psi4")
np   = pyimport("numpy") # used only to cast to Psi4 arrays
using TensorOperations: @tensor
using LinearAlgebra: Diagonal, Hermitian, eigen, tr
using Printf: @printf
```

Next, using what you learned in the previous tutorial module, set the following  Psi4  and molecule options.

Memory & Output specifications:
- Give 500 Mb of memory to Psi4
- Set Psi4 output file to "output.dat"
- Set a variable `numpy_memory` to an acceptable amount of available memory for the working computer to use for storing tensors

Molecule definition:
- Define the "physicist's water molecule" (O-H bond length = 1.1 Angstroms, HOH bond angle = 104 degrees)
- Molecular symmetry C1

Computation options:
- basis set cc-pVDZ
- SCF type PK
- Energy convergence criterion to 0.00000001

```julia
# ==> Set Basic Psi4 Options <==
# Memory specification
psi4.set_memory(Int(5e8))
numpy_memory = 2

# Set output file
psi4.core.set_output_file("output.dat", false)

# Define Physicist's water -- don't forget C1 symmetry!
mol = psi4.geometry("""
O
H 1 1.1
H 1 1.1 2 104
symmetry c1
""")

# Set computation options
psi4.set_options(Dict("basis" => "cc-pvdz",
                      "scf_type" => "pk",
                      "e_convergence" => 1e-8))
```

      Memory set to 476.837 MiB by Python driver.

Since we will be writing our own, iterative RHF procedure, we will need to define options that we can use to tweak our convergence behavior.  For example, if something goes wrong and our SCF doesn't converge, we don't want to spiral into an infinite loop.  Instead, we can specify the maximum number of iterations allowed, and store this value in a variable called `maxiter`.  Here are some good default options for our program:
~~~python
MAXITER = 40
E_conv = 1.0e-6
~~~
These are by no means the only possible values for these options, and it's encouraged to try different values and see for yourself how different choices affect the performance of our program.  For now, let's use the above as our default.

```julia
# ==> Set default program options <==
# Maximum SCF iterations
MAXITER = 40
# Energy convergence criterion
E_conv = 1.0e-6;
```

Before we can build our Fock matrix, we'll need to compute the following static one- and two-electron quantities:

- Electron repulsion integrals (ERIs) **I** between our AOs
- Overlap matrix **S**
- Core Hamiltonian matrix **H**

Fortunately for us, we can do this using the machinery in  Psi4.  In the first module, you learned about `psi4.core.Wavefunction` and `psi4.core.MintsHelper` classes.  In the cell below, use these classes to perform the following:

1. Create Class Instances

    a. Build a wavefunction for our molecule and basis set

    b. Create an instance of the `MintsHelper` class with the basis set for the wavefunction

2. Build overlap matrix, **S**

    a. Get the AO overlap matrix from `MintsHelper`, and cast it into a Julia array

    b. Get the number of AO basis functions and number of doubly occupied orbitals from S and the wavefunciton

3. Compute ERI Tensor, **I**

    a. Get ERI tensor from `MintsHelper`, and cast it into a Julia array

4. Build core Hamiltonian, **H**

    a. Get AO kinetic energy matrix from `MintsHelper`, and cast it into a Julia array

    b. Get AO potential energy matrix from `MintsHelper`, and cast it into a Julia array

    c. Build core Hamiltonian from kinetic & potential energy matrices

```julia
# ==> Compute static 1e- and 2e- quantities with Psi4 <==
# Class instantiation
wfn = psi4.core.Wavefunction.build(mol, psi4.core.get_global_option("basis"))
mints = psi4.core.MintsHelper(wfn.basisset())

# Overlap matrix
S = np.asarray(mints.ao_overlap()) # we only need a copy

# Number of basis Functions & doubly occupied orbitals
nbf = size(S)[1]
ndocc = wfn.nalpha()

println("Number of occupied orbitals: ", ndocc)
println("Number of basis functions: ", nbf)

# Memory check for ERI tensor
I_size = nbf^4 * 8.e-9
println("\nSize of the ERI tensor will be $I_size GB.")
memory_footprint = I_size * 1.5
if I_size > numpy_memory
    psi4.core.clean()
    throw(OutOfMemoryError("Estimated memory utilization ($memory_footprint GB) exceeds " *
                           "allotted memory limit of $numpy_memory GB."))
end

# Build ERI Tensor
I = np.asarray(mints.ao_eri()) # we only need a copy

# Build core Hamiltonian
T = np.asarray(mints.ao_kinetic()) # we only need a copy
V = np.asarray(mints.ao_potential()) # we only need a copy
H = T + V;
```

    Number of occupied orbitals: 5
    Number of basis functions: 24

    Size of the ERI tensor will be 0.0026542080000000003 GB.

The Roothan equations

$${\bf FC} = {\bf SC\epsilon}$$

are only *pseudo*-eigenvalue equations due to the presence of the overlap matrix **S** on the right hand side of the equation.  Normally, the AO basis set will not be orthonormal, so the overlap matrix **S** will not be unity and therefore cannot be ignored.  Let's check to see whether our AO basis is orthonormal:

```julia
# ==> Inspecting S for AO orthonormality <==
hope = S ≈ Diagonal(ones(size(S)[1]))
println("\nDo we have any hope that our AO basis is orthonormal? ", hope)
```

    Do we have any hope that our AO basis is orthonormal? false

Just as we'd expected -- looks like we can't ignore the AO overlap matrix.  Therefore, the Fock matrix **F** cannot simply be diagonalized to solve for the orbital coefficient matrix **C**.  There is still hope, however!  We can overcome this issue by transforming the AO basis so that all of our basis functions are orthonormal.  In other words, we seek a matrix **A** such that the transformation

$${\bf A}^{\dagger}{\bf SA} = {\bf 1}$$

One method of doing this is called *symmetric orthogonalization*, which lets ${\bf A} = {\bf S}^{-1/2}$.  Then,

$${\bf A}^{\dagger}{\bf SA} = {\bf S}^{-1/2}{\bf SS}^{-1/2} = {\bf S}^{-1/2}{\bf S}^{1/2} = {\bf S}^0 = {\bf 1},$$

and we see that this choice for **A** does in fact yield an orthonormal AO basis.  In the cell below, construct this transformation matrix using  Psi4's built-in `Matrix` class member function `power()` just like the following:
~~~python
A = mints.ao_overlap()
A.power(-0.5, 1.e-16)
A = np.asarray(A)
~~~

```julia
# ==> Construct AO orthogonalization matrix A <==
A = mints.ao_overlap()
A.power(-0.5, 1.e-16) # ≈ Julia's A^(-0.5) after psi4view()
A = np.asarray(A) # we only need a copy

# Check orthonormality
S_p = A * S * A
new_hope = S ≈ Diagonal(ones(size(S)[1]))

if new_hope
    println("There is a new hope for diagonalization!")
else
    println("Whoops...something went wrong. Check that you've correctly built the transformation matrix.")
    @show sum(S_p) - tr(S_p)
end
```

    Whoops...something went wrong. Check that you've correctly built the transformation matrix.
    sum(S_p) - tr(S_p) = -3.552713678800501e-15

    -3.552713678800501e-15

The drawback of this scheme is that we would now have to either re-compute the ERI and core Hamiltonian tensors in the newly orthogonal AO basis, or transform them using our **A** matrix (both would be overly costly, especially transforming **I**).  On the other hand, substitute ${\bf C} = {\bf AC}'$ into the Roothan equations:

\begin{align}
{\bf FAC'} &= {\bf SAC}'{\bf \epsilon}\\
{\bf A}^{\dagger}({\bf FAC}')&= {\bf A}^{\dagger}({\bf SAC}'){\bf \epsilon}\\
({\bf A}^{\dagger}{\bf FA}){\bf C}'&= ({\bf A}^{\dagger}{\bf SA}){\bf C}'{\bf \epsilon}\\
{\bf F}'{\bf C}' &= {\bf 1C}'{\bf \epsilon}\\
{\bf F}'{\bf C}' &= {\bf C}'{\bf \epsilon}\\
\end{align}

Clearly, we have arrived at a canonical eigenvalue equation.  This equation can be solved directly for the transformed orbital coefficient matrix ${\bf C}'$ by diagonalizing the transformed Fock matrix, ${\bf F}'$, before transforming ${\bf C}'$ back into the original AO basis with ${\bf C} = {\bf AC}'$.

Before we can get down to the business of using the Fock matrix **F** to compute the RHF energy, we first need to compute the orbital coefficient **C** matrix.  But, before we compute the **C** matrix, we first need to build **F**.  Wait...hold on a second.  Which comes first, **C** or **F**?  Looking at the Roothan equations more closely, we see that that both sides depend on the **C** matrix, since **F** is a function of the orbitals:

$${\bf F}({\bf C}){\bf C} = {\bf SC\epsilon}\,;\;\;F_{\mu\nu} = H_{\mu\nu} + 2(\mu\,\nu\mid\lambda\,\sigma)C_{\sigma i}C_{\lambda i} - (\mu\,\lambda\,\mid\nu\,\sigma)C_{\sigma i}C_{\lambda i}.$$

Therefore technically, *neither* **F** nor **C** can come first!  In order to proceed, we instead begin with a *guess* for the Fock matrix, from which we obtain a guess at the **C** matrix.  Without orbital coefficients (and therefore without electron densities), the most logical starting point for obtaining a guess at the Fock matrix is to begin with the only component of **F** that does *not* involve densities: the core Hamiltonian, **H**.  Below, using the `eigen()` function, and forcing hermitianess with `Hermitian()`, obtain coefficient and density matrices using the core guess:

1. Obtain ${\bf F}'$ by transforming the core Hamiltonian with the ${\bf A}$ matrix
2. Diagonalize the transformed Fock matrix for $\epsilon$ and ${\bf C}'$
3. Use doubly-occupied slice of coefficient matrix to build density matrix

```julia
# ==> Compute C & D matrices with CORE guess <==
# Transformed Fock matrix
F_p = A * H * A

# Diagonalize F_p for eigenvalues & eigenvectors with NumPy
e, C_p = eigen(Hermitian(F_p))

# Transform C_p back into AO basis
C = A * C_p

# Grab occupied orbitals
C_occ = C[:, 1:ndocc]

# Build density matrix from occupied orbitals
D = C_occ * C_occ' ;
```

The final quantity we need to compute before we can proceed with our implementation of the SCF procedure is the Born-Oppenheimer nuclear repulsion energy, $E^{\rm BO}_{\rm nuc}$.  We could use the expression given above in $\S$1, however we can also obtain this value directly from  Psi4's `Molecule` class.  In the cell below, compute the nuclear repulsion energy using either method.

```julia
# ==> Nuclear Repulsion Energy <==
E_nuc = mol.nuclear_repulsion_energy()
```

    8.002366482173422

Within each SCF iteration, we'll have to perform a number of tensor contractions when building the Fock matrix, computing the total RHF energy, and performing several transformations.  Since the computational expense of this process is related to the number of unique indices, the most intensive step of computing the total electronic energy will be performing the four-index contractions corresponding to building Coulomb and Exchange matrices **J** and **K**, with elements

\begin{align}
J[D_{\lambda\sigma}]_{\mu\nu} &= (\mu\,\nu\mid\lambda\,\sigma)D_{\lambda\sigma}\\
K[D_{\lambda\sigma}]_{\mu\nu} &= (\mu\,\lambda\mid\nu\,\sigma)D_{\lambda\sigma},
\end{align}

when building the Fock matrix.  Fortunately, once **J** and **K** have been built, the Fock matrix may be computed as a simple matrix addition, instead of element-wise:

$$ {\bf F} = {\bf H} + 2{\bf J} - {\bf K} = {\bf H} + {\bf G}.$$

Formation of the **J** and **K** matrices will be the most expensive step of the RHF procedure, scaling with respect to the number of AOs as ${\cal O}(N^4)$.  Strategies for building these marices efficiently, as well as different methods for handling these tensor contractions, will be discussed in greater detail in tutorials 2c and 2d in this module, respectively.

Let's now write our SCF iterations according to the following algorithm:

#### Algorithm 1: SCF Iteration
for `scf_iter` less than `MAXITER`, do:
1. Build Fock matrix
    - Build the two-electron Coulomb & Exchange matrix **G**
    - Form the Fock matrix
2. RHF Energy
    - Compute total RHF energy
    - If change in RHF energy less than `E_conv`, break
    - Save latest RHF energy as `E_old`
3. Compute new orbital guess
    - Transform Fock matrix to orthonormal AO basis
    - Diagonalize ${\bf F}'$ for $\epsilon$ and ${\bf C}'$
    - Back transform ${\bf C}'$ to AO basis
    - Form **D** from occupied orbital slice of **C**

```julia
# ==> SCF Iterations <==
# Output and pre-iteration energy declarations
SCF_E = let SCF_E = 0.0, E_old = 0.0, D = D

   print("==> Starting SCF Iterations <==\n")

   # Begin Iterations
   for scf_iter in 1:MAXITER
       # Build Fock matrix
       @tensor G[p,q] := (2I[p,q,r,s] - I[p,r,q,s]) * D[r,s]
       F = H + G

       # Compute RHF energy
       SCF_E = tr((H + F) * D) + E_nuc
       @printf("SCF Iteration %3d: Energy = %4.16f dE = %1.5e \n",scf_iter, SCF_E, SCF_E - E_old)

       # SCF Converged?
       if abs(SCF_E - E_old) < E_conv
           break
       end
       E_old = SCF_E

       # Compute new orbital guess
       F_p =  A * F * A
       e, C_p = eigen(Hermitian(F_p))
       C = A * C_p
       C_occ = C[:, 1:ndocc]
       D = C_occ * C_occ'

       # MAXITER exceeded?
       if scf_iter == MAXITER
           psi4.core.clean()
           throw(MethodError("Maximum number of SCF iterations exceeded."))
       end
   end
   SCF_E # return RHF SCF energy
end

# Post iterations
println("\nSCF converged.")
println("Final RHF Energy: $SCF_E [Eh]")
```

    ==> Starting SCF Iterations <==
    SCF Iteration   1: Energy = -68.9800327333871053 dE = -6.89800e+01
    SCF Iteration   2: Energy = -69.6472544393141675 dE = -6.67222e-01
    SCF Iteration   3: Energy = -72.8403031079928667 dE = -3.19305e+00
    SCF Iteration   4: Energy = -72.8948839248304239 dE = -5.45808e-02
    SCF Iteration   5: Energy = -74.1207806666203908 dE = -1.22590e+00
    SCF Iteration   6: Energy = -74.8671819750254031 dE = -7.46401e-01
    SCF Iteration   7: Energy = -75.4149088033799728 dE = -5.47727e-01
    SCF Iteration   8: Energy = -75.7076722235638897 dE = -2.92763e-01
    SCF Iteration   9: Energy = -75.8605242159064517 dE = -1.52852e-01
    SCF Iteration  10: Energy = -75.9314385293936596 dE = -7.09143e-02
    SCF Iteration  11: Energy = -75.9640243989531712 dE = -3.25859e-02
    SCF Iteration  12: Energy = -75.9784292086707467 dE = -1.44048e-02
    SCF Iteration  13: Energy = -75.9848169970782976 dE = -6.38779e-03
    SCF Iteration  14: Energy = -75.9876128367924082 dE = -2.79584e-03
    SCF Iteration  15: Energy = -75.9888408939168016 dE = -1.22806e-03
    SCF Iteration  16: Energy = -75.9893777546510591 dE = -5.36861e-04
    SCF Iteration  17: Energy = -75.9896129374999703 dE = -2.35183e-04
    SCF Iteration  18: Energy = -75.9897157734858411 dE = -1.02836e-04
    SCF Iteration  19: Energy = -75.9897607856073023 dE = -4.50121e-05
    SCF Iteration  20: Energy = -75.9897804728111623 dE = -1.96872e-05
    SCF Iteration  21: Energy = -75.9897890875960513 dE = -8.61478e-06
    SCF Iteration  22: Energy = -75.9897928560694425 dE = -3.76847e-06
    SCF Iteration  23: Energy = -75.9897945049109751 dE = -1.64884e-06
    SCF Iteration  24: Energy = -75.9897952262384706 dE = -7.21327e-07

    SCF converged.
    Final RHF Energy: -75.98979522623847 [Eh]

Congratulations! You've written your very own Restricted Hartree-Fock program!  Finally, let's check your final RHF energy against  Psi4:

```julia
# Compare to Psi4
SCF_E_psi = psi4.energy("SCF")
psi4.compare_values(SCF_E_psi, SCF_E, 6, "SCF Energy")
```

    	SCF Energy........................................................PASSED

    true

## References
1. [[Szabo:1996](http://store.doverpublications.com/0486691861.html)] A. Szabo and N. S. Ostlund, *Modern Quantum Chemistry*, Introduction to Advanced Electronic Structure Theory. Courier Corporation, 1996.
2. [[Levine:2000](https://books.google.com/books?id=80RpQgAACAAJ&dq=levine%20quantum%20chemistry%205th%20edition&source=gbs_book_other_versions)] I. N. Levine, *Quantum Chemistry*. Prentice-Hall, New Jersey, 5th edition, 2000.
3. [[Helgaker:2000](https://books.google.com/books?id=lNVLBAAAQBAJ&pg=PT1067&dq=helgaker+molecular+electronic+structure+theory&hl=en&sa=X&ved=0ahUKEwj37I7MkofUAhWG5SYKHaoPAAkQ6AEIKDAA#v=onepage&q=helgaker%20molecular%20electronic%20structure%20theory&f=false)] T. Helgaker, P. Jorgensen, and J. Olsen, *Molecular Electronic Structure Theory*, John Wiley & Sons Inc, 2000.


## Devito CFD Tutorial series

The following series of notebook tutorials will demonstrate the use of Devito and it's SymPy-based API to solve a set of classic examples from Computational Fluid Dynamics (CFD). The tutorials are based on the excellent tutorial series _CFD Python: 12 steps to Navier-Stokes_ by Lorena Barba and focus on the implementation with Devito rather than pure CFD or finite difference theory. For a refresher on how to implement 2D finite difference solvers for CFD problems, please see the original tutorial series here: <br>
http://lorenabarba.com/blog/cfd-python-12-steps-to-navier-stokes/

### Example 1: Linear convection in 2D

Lets start with a simple 2D convection example - step 5 in the original blog. This will already allow us to demonstrate a lot about the use of Devito's symbolic data objects and how to use them to build a simple operator directly from the symbolic notation of the equation. The governing equation we will implement in this tutorial is:

$$\frac{\partial u}{\partial t}+c\frac{\partial u}{\partial x} + c\frac{\partial u}{\partial y} = 0$$

In order to implement this equation we first discretize it using forward differences in time and backward differences in space. Just as the original tutorial, we will use $u_{i,j}^n$ to denote a finite difference stencil point with $i$ and $j$ denoting spatial indices and $n$ denoting the time index. So, after re-arragning the discretized equation for the forward stencil point in time we get

$$u_{i,j}^{n+1} = u_{i,j}^n-c \frac{\Delta t}{\Delta x}(u_{i,j}^n-u_{i-1,j}^n)-c \frac{\Delta t}{\Delta y}(u_{i,j}^n-u_{i,j-1}^n)$$

Using this, we can start deriving the computational stencil for this equation. Let's first look at the original _pure Python_ implementation of the linear convection flow - but first we import our tools and define some parameters:

```python
from examples.cfd import plot_field, init_hat
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 81
ny = 81
nt = 100
c = 1.
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
print("dx %s, dy %s" % (dx, dy))
sigma = .2
dt = sigma * dx
```

    dx 0.025, dy 0.025

**A small note on style:** Throughout this tutorial series we will use utility functions to plot the various 2D functions and data sets we deal with. These are all taken from the original tutorial series, but have been slightly modified for our purposes. One of the differences readers might find is that the original series uses _(y, x)_ indexing for 2d data arrays, whereas many of the examples have been adapted to use _(x, y)_ notation in our tutorials.

So, let's start by creating a simple 2D function and initialising it with a "hat function". We will use that initialisation function a lot, so it comes from our utility scripts:

```python
#NBVAL_IGNORE_OUTPUT

# Create field and assign initial conditions
u = np.empty((nx, ny))
init_hat(field=u, dx=dx, dy=dy, value=2.)

# Plot initial condition
plot_field(u)
```

Now we can repeat the _pure NumPy_ solve from the original tutorial, where we use NumPy array operations to speed up the computation. Note that we skip the derivation of the stencil used to implement our convection equation, as we are going to walk though this process using the Devito API later in this tutorial.

```python
# Repeat initialisation, so we can re-run the cell
init_hat(field=u, dx=dx, dy=dy, value=2.)

for n in range(nt + 1):
    # Copy previous result into a new buffer
    un = u.copy()

    # Update the new result with a 3-point stencil
    u[1:, 1:] = (un[1:, 1:] - (c * dt / dx * (un[1:, 1:] - un[1:, :-1])) -
                              (c * dt / dy * (un[1:, 1:] - un[:-1, 1:])))

    # Apply boundary conditions
    u[0, :] = 1.
    u[-1, :] = 1.
    u[:, 0] = 1.
    u[:, -1] = 1.
```

```python
#NBVAL_IGNORE_OUTPUT

# A small sanity check for auto-testing
assert (u[45:55, 45:55] > 1.8).all()
u_ref = u.copy()

plot_field(u)
```

Hooray, the wave moved! Now, this little example is already very concise from a notational point of view and it teaches us quite a few key points about how to perform finite difference stencil computation via NumPy:
* Due to the backward differencing scheme in space (more later) we use only three stencil point in this example:  $u^{n}_{i, j}$, $u^{n}_{i-1, j}$ and $u^{n}_{i, j-1}$. These can be identified in the code through the array indices and correspond to `un[1:, 1:]`, `un[:-1, 1:]` and `un[1:, :-1]` respectively.
* Two buffers for array data are used throughout: `un[...]` is read from, while `u[...]` is updated, where the line `un = u.copy()` performs a deep copy of the field to switch buffers between timesteps. Note that in some other finite difference tutorials the cost of this copy operation is sometimes amortised by using two pre-allocated buffers and switching the indices of them explicitly.
* The final four lines within the loop code show us how to implmement simple Dirichlet boundary condition by simply setting a value on the outermost rows and columns of our cartesian grid.

You may have noticed that the hat has not only moved to a different location, but has also changed its shape into a smooth bump. This is a little surprising, as the correct solution of the convection equation would be movement without shape change. The smooth shape is caused by numerical diffusion, a well-known limitation of low-order finite difference schemes. We will discuss this issue and some solutions later in this tutorial.

#### Devito implementation
Now we want to re-create the above example via a Devito operator. To do this, we can start by defining our computational grid and creating a function `u` as a symbolic `devito.TimeFunction`. The core thing to note here is that this is one of Devito's symbolic function, which have a dual role in the creation of finite difference solvers:
* They behave symbolically like `sympy.Function` objects, so that we can construct derivatives and use them in symbolic expressions, thus inheriting all the power of automated symbolic manipulation that SymPy provides.
* They act as containers for user data by providing a `.data` property that wraps automatically allocated memory space in a neat NumPy array.

The particular `TimeFunction` type that we will declare our variable $u$ as in this case is aware of the fact that we will want to implement a timestepping algorithm with it. So the object `u` will declare two buffers of shape `(nx, ny)` for us, as defined by the `Grid` object, and present them as `u.data[0]` and `u.data[1]`. Let's fill the initial buffer with some data and look at it:

```python
#NBVAL_IGNORE_OUTPUT
from devito import Grid, TimeFunction

grid = Grid(shape=(nx, ny), extent=(2., 2.))
u = TimeFunction(name='u', grid=grid)
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)

plot_field(u.data[0])
```

Nice. Now we can look at deriving our 3-point stencil using the symbolic capabilities given to our function $u$ by SymPy. For this we will first construct our derivative terms in space and time. For the forward derivative in time we can easily use Devito's shorthand notation `u.dt` to denote the first derivative in time and `u.dxl` and `u.dyl` to denote the space derivatives. Note that the `l` means were using the "left" or backward difference here to adhere to the discretization used in the original tutorials.

From the resulting terms we can then create a `sympy.Equation` object that contains the fully discretised equation, but from a neat high-level notation, as shown below.

```python
from sympy import Eq

eq = Eq(u.dt + c*u.dxl + c*u.dyl)

print(eq)
```

    Eq(1.0*u(t, x, y)/h_y - 1.0*u(t, x, y - h_y)/h_y + 1.0*u(t, x, y)/h_x - 1.0*u(t, x - h_x, y)/h_x - u(t, x, y)/dt + u(t + dt, x, y)/dt, 0)

The above step resulted in a fully discretised version of our equation, which includes place-holder symbols for the spacing in time (`s`) and space (`h`). These symbols are based on an internal convention and will later be replaced when we build an operator.

But before we can build an operator, we first need to change our discretised expression so that we are updating the forward stencil point in our timestepping scheme - Devito provides another short-hand notation for this: `u.forward`. For the actual symoblic reordering, SymPy comes to the rescue with the `solve` utility that we can use to re-organise our equation.

```python
from sympy import solve

stencil = solve(eq, u.forward)[0]

print(stencil)
```

    (dt*h_x*(-u(t, x, y) + u(t, x, y - h_y)) + dt*h_y*(-u(t, x, y) + u(t, x - h_x, y)) + h_x*h_y*u(t, x, y))/(h_x*h_y)

The careful reader will note that this is now the equivalent symbolic expression to the RHS term of the NumPy code we showed earlier - only with `dx` and `dy` denoted as `h` and `dt` denoted as `s`, while `u(t, x, y)`, `u(t, x - h, y)` and `u(t, x, y - h)` denote the equivalent of $u^{n}_{i, j}$, $u^{n}_{i-1, j}$ and $u^{n}_{i, j-1}$ respectively.

We can now use this stencil expression to create an operator to apply to our data object:

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

# Reset our initial condition in both buffers.
# This is required to avoid 0s propagating into
# our solution, which has a background value of 1.
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=u.data[1], dx=dx, dy=dy, value=2.)

# Create an operator that updates the forward stencil point
op = Operator(Eq(u.forward, stencil))

# Apply the operator for a number of timesteps
op(time=nt+1, dt=dt)

plot_field(u.data[0, :, :])

# Some small sanity checks for the testing framework
assert (u.data[0, 45:55, 45:55] > 1.8).all()
assert np.allclose(u.data[0], u_ref, rtol=3.e-2)
```

Great, that looks to have done the same thing as the original NumPy example, so we seem to be doing something right, at least.

**A note on performance:** During the code generation phase of the previous operatorm Devito has created some log output with information about two optimisation steps: The DSE (Devito Symbolics Engine) and the DLE (Devito Loop Engine). We can ignore these for now, because our exmaple is tiny - but for large runs where performance matters, these are the engines that make the Devito kernel run very fast in comparison to raw Python/NumPy.

Now, despite getting a correct looking result, there is still one problem with the above operator: It doesn't set any boundary conditions as part of the time loop. We also note that the operator includes a time loop, but at this point Devito doesn;t actually provide any language constructs to explicitly define different types of boundary conditions (Devito is probably still a kind of prototype at this point). Luckily though, Devito provides a backdoor for us to insert custom expression in the so-called "indexed" or "low-level" API that allow us to encode the Dirichlet boundary condition of the original example.

#### The "indexed" or low-level API
The `TimeFunction` field we created earlier behaves symbolically like a `sympy.Function` object with the appropriate indices, eg. `u(t, x, y)`. If we take a simple first-order derivative of that we have a term that includes the spacing variable `h`, which Devito uses as the default for encoding $dx$ or $dy$. For example, `u.dx` simply expands to `-u(t, x, y)/h + u(t, x + h, y)/h`.

Now, when the `Operator` creates explicit C-code from that expression, it at some point "lowers" that expression by resolving explicit data accesses (or indices) into our grid by transforming it into a `sympy.Indexed` object. During this process all occurences of `h` in data accesses get replaced with integers, so that the expression now looks like `-u[t, x, y]/h + u[t, x + 1, y]/h`. This is the "indexed" notation and we can create custom expression of the same kind by using the `u.indexed` property with explicit indices in square-bracket notation. These custom expressions can then be injected them into our operator like this:

```python
#NBVAL_IGNORE_OUTPUT

# Reset our data field and ICs in both buffers
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=u.data[1], dx=dx, dy=dy, value=2.)

# For defining BCs, we want to explicitly set rows/columns in our field
# We can use Devito's "indexed" notation to do this:
x, y = grid.dimensions
t = grid.stepping_dim
bc_left = Eq(u.indexed[t + 1, 0, y], 1.)
bc_right = Eq(u.indexed[t + 1, nx-1, y], 1.)
bc_top = Eq(u.indexed[t + 1, x, ny-1], 1.)
bc_bottom = Eq(u.indexed[t + 1, x, 0], 1.)

# Now combine the BC expressions with the stencil to form operator
expressions = [Eq(u.forward, stencil)]
expressions += [bc_left, bc_right, bc_top, bc_bottom]
op = Operator(expressions=expressions, dle=None, dse=None)  # <-- Turn off performance optimisations
op(time=nt+1, dt=dt)

plot_field(u.data[0, :, :])

# Some small sanity checks for the testing framework
assert (u.data[0, 45:55, 45:55] > 1.8).all()
assert np.allclose(u.data[0], u_ref, rtol=3.e-2)
```

You might have noticed that we used the arguments `dle=None` and `dse=None` in the creation of the previous operator. This suppresses the various performance optimisation steps in the code-generation pipeline, which makes the auto-generated C code much easier to look at. So, for the brave, let's have a little peek under the hood...

```python
print(op.ccode)
```

    #define _POSIX_C_SOURCE 200809L
    #include "stdlib.h"
    #include "math.h"
    #include "sys/time.h"

    struct profile
    {
      double section_0;
      double section_1;
      double section_2;
    } ;

    int Kernel(const float dt, const float h_x, const float h_y, float *restrict u_vec, const int t_size, const int t_s, const int t_e, const int time_size, const int time_s, const int time_e, const int x_size, const int x_s, const int x_e, const int y_size, const int y_s, const int y_e, void *_timings)
    {
      float (*restrict u)[x_size][y_size] __attribute__((aligned(64))) = (float (*)[x_size][y_size]) u_vec;
      struct profile *timings = (struct profile*) _timings;
      for (int time = t_s, t0 = (time)%(2), t1 = (time + 1)%(2); time < t_e - 1; time += 1, t0 = (time)%(2), t1 = (time + 1)%(2))
      {
        struct timeval start_section_0, end_section_0;
        gettimeofday(&start_section_0, NULL);
        for (int x = x_s + 1; x < x_e; x += 1)
        {
          for (int y = y_s + 1; y < y_e; y += 1)
          {
            u[t1][x][y] = (dt*h_x*(-u[t0][x][y] + u[t0][x][y - 1]) + dt*h_y*(-u[t0][x][y] + u[t0][x - 1][y]) + h_x*h_y*u[t0][x][y])/(h_x*h_y);
          }
        }
        gettimeofday(&end_section_0, NULL);
        timings->section_0 += (double)(end_section_0.tv_sec-start_section_0.tv_sec)+(double)(end_section_0.tv_usec-start_section_0.tv_usec)/1000000;
        struct timeval start_section_1, end_section_1;
        gettimeofday(&start_section_1, NULL);
        for (int y = y_s + 1; y < y_e; y += 1)
        {
          u[t1][0][y] = 1.00000000000000F;
          u[t1][80][y] = 1.00000000000000F;
        }
        gettimeofday(&end_section_1, NULL);
        timings->section_1 += (double)(end_section_1.tv_sec-start_section_1.tv_sec)+(double)(end_section_1.tv_usec-start_section_1.tv_usec)/1000000;
        struct timeval start_section_2, end_section_2;
        gettimeofday(&start_section_2, NULL);
        for (int x = x_s + 1; x < x_e; x += 1)
        {
          u[t1][x][80] = 1.00000000000000F;
          u[t1][x][0] = 1.00000000000000F;
        }
        gettimeofday(&end_section_2, NULL);
        timings->section_2 += (double)(end_section_2.tv_sec-start_section_2.tv_sec)+(double)(end_section_2.tv_usec-start_section_2.tv_usec)/1000000;
      }
      return 0;
    }