One-day Stock Price Forecasting Using Deep Neural Netrowks

Table of Contents

Problem Description

Data Preparation

Model

Training

Results

Source code

Problem Description

Deep Learning neural networks are capable of modeling a process by learning from a given dataset. That is, a neural network is a function $\hat{y}=f(x|W)$ that can be used to approximate most fucntions using a set of trainable parameters.

If a given neural netrowk can be trained to mimic the behaviour of some time series $s(t)$ we can use such network to make predictions for a given time. More formally, we want to perform univariate one-step forecasting on a dataset of daily close prices to capture the behaviour of a particular stock.

\begin{align}
\hat{s}(t) = f(s(t-1),s(t-2),...,s(t-l)|W)\\
\end{align}

Here, $W$ is the set of trainable weights that are used inside the network to map a feature vector to a target (input to output). We represent the daily close prices as follows,

\begin{align}
d = \{(\mathbf{x}_i,y_i)\}^{n-1}_{i=0}\\
\end{align}

The dataset has $n$ tuples $(\mathbf{x}_i,y_i)$ representing the $\textit{i-th}$ feature vector $\mathbf{x}_i=[s(t-1), s(t-2), ..., s(t-l)]$ and the scalar training sample $y_i=s(t)$. The parameter $l$ controls the number of past samples included in the forecast.

Note that $d$ is not the original time series $s(t)$, we first have to re-frame the raw data set to one suitable for a supervised learning problem by transforming the univariate time series into a multivariate one via time delay embedding,

\begin{equation}
X =
\begin{bmatrix}
s_0 & s_1 & s_2 & \cdots & s_{l-1}\\
s_1 & s_2 & s_3 &\cdots & s_{l}\\
s_2 & s_3 & s_4 & \cdots & s_{l+1}\\
\vdots & \vdots & \vdots & \ddots & \vdots \\
s_{m-l} & s_{m-l+1} & s_{m-l+2} & \cdots & s_{m-1}\\
\end{bmatrix}
\end{equation}

Here, $m$ is the last sample in the original data set and the feature vectors are the rows of $X$. Simmlarly, the targets are the following scalar entries,

\begin{equation}
Y =
\begin{bmatrix}
s_l\\
s_{l+1}\\
s_{l+2}\\
\vdots\\
s_{m}
\end{bmatrix}
\end{equation}

Data Preparation

We need to make the supervised learning dataset and create a Dataset class to work with DataLoader.

```python
class DataSupervised(Dataset):
    '''
        Custom dataset to work with DataLoader for supervised learning.

        file_name (str): Path to csv file.
        target_cols (int): Number of steps (forecasts), one by default (last column).
        train (bool): Train (odd samples) or test split (even samples), True by default.
    '''

    def __init__(self, file_name, target_cols=1, train=True):

        stock_supervised = pd.read_csv(file_name).values
        X_train, X_test = train_test_split(stock_supervised,test_size=0.2)
        if train:
            self.X = torch.FloatTensor(X_train[:,:-target_cols])
            self.Y = torch.FloatTensor(X_train[:,-target_cols])
            if target_cols == 1:
                self.Y = self.Y.unsqueeze(1)
        else:
            self.X = torch.FloatTensor(X_test[:,:-target_cols])
            self.Y = torch.FloatTensor(X_test[:,-target_cols])
            if target_cols == 1:
                self.Y = self.Y.unsqueeze(1)

        self.n_samples = self.X.shape[0]

    def __getitem__(self, index):
        return self.X[index], self.Y[index]

    def __len__(self):
        return self.n_samples
```

Model

One of the simplest architectures for a multi-dimensional inputs is the [multi-layer perceptron](https://en.wikipedia.org/wiki/Multilayer_perceptron). It is composed of an input layer followed by a variable number of fully connected layers and nodes, and the output layer.

The number of neurons in the input layer corresponds to the number of features the model uses, while the number of nodes in the output layer tells us how many samples we are forecasting. The number of neurons in the hidden layer translates to the complexity of the model. A model with too many hidden neurons could result in a model that is too complex for the dataset and thus lead to overfitting. In contrast, a model with too few hidden neurons can fail to capture the complexity of the data and result in underfitting.

We define our neural network as a class that allows for custom number of hidden layers, number of neurons per hidden layer, and choice of non-linear transformation in the hidden layer.

```python
class ANN(nn.Module):
    """Artificial Neural Network

        MLP with tanh activation function for the hidden layers and linear transformation
        for the output layer by default.

        Layers -- (list) Numbers of neurons in each layer.
    """

    def __init__(self, Layers):
        super(ANN, self).__init__()
        self.hidden = nn.ModuleList()

        for input_size, output_size in zip(Layers, Layers[1:]):
            linear = nn.Linear(input_size, output_size)
            self.hidden.append(linear)

    def forward(self, x):
        layers = len(self.hidden)
        for (layer, linear_transform) in zip(range(layers), self.hidden):
            if layer < layers - 1:
                x = torch.tanh(linear_transform(x))
            else:
                x = linear_transform(x)
        return x
```

*Multi-dimensional input fully connected neural network with $p$ perceptrons or neurons in a single hidden layer. The nodes represent the inputs, activations or outputs and the edges represent the weights and biases. The biases are assumed to be zero for simplicity. The yellow circles represent a linear transformation while the blue rectangle represents a non-linear transformation (activation function).*

Training

To train the network there are a couple of things we need to define:
* The loss function
* How the data gets fed to the network

 we are modeling a time-series via non-linear regression, thus using the mean squared error (MSE) is a good starting point to measure the performance of the network.

\begin{align}
\mathit{MSE}:=\frac{1}{n} \sum_{i=0}^{n-1} (\mathbf{y}_{i}-\hat{\mathbf{y}}_{i})^2
\end{align}

Where $\hat{\mathbf{y}}_{i}$ is the i-th estimated value. Notice that the definition above averages over all training samples for a given epoch.

For very large datasets it is sometimes conveient to feed the training samples in batches to avoid computing the gradient over all samples. This approximationg approaches the true gradient as the batch size tends to the whole dataset. As a consequence the MSE is computing for a given batch.

Lastly, we shuffle the training samples at each epoc to increase the robustness of the model. This ensures that the model is not fed the same batch at later epochs

[What is batch size in neural networks](https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network)

[Why should the data be shuffled for machine learning tasks](https://datascience.stackexchange.com/questions/24511/why-should-the-data-be-shuffled-for-machine-learning-tasks)

```python
def train_series(train_loader, model, criterion, optimizer, epochs=10, test_loader=None, display_batch=False):
    """Train and test (optional) a time-series model, returns the loss at a given epoch.

    train_loader -- DataLoader object with the training dataset.
    model -- Neural Network to be trained.
    criterion -- Loss function.
    optimizer -- optimization algorithm to update the network weights.
    epochs -- Number of forward and backward passes on the whole dataset.
    test_loader -- DataLoader object with the test dataset (default None).
    display_batch -- Display epoch, bactch index and batch length (default False).
    """

    model.train()
    results = {'training loss': [], 'validation error': []}   # loss at a given epoch
    for epoch in range(epochs):

        total = 0 # training loss for every epoch
        for batch_idx, (x, y) in enumerate(train_loader):
            if display_batch:
              print('epoch {}, batch idx {} , batch len {}'.format(epoch, batch_idx, len(y)))
            optimizer.zero_grad()
            loss = criterion(model(x), y)
            loss.backward()
            optimizer.step()
            total += loss.item()  # cummulative loss/batch
        results['training loss'].append(total/batch_idx) # ~ loss over all training samples

        if test_loader is not None:
            results['validation error'].append(test_series(test_loader, model, criterion))

    return results
```

We also define the auxiliary method for computing the error on a test set.

```python
def test_series(test_loader, model, criterion):
    """ test a a time-series model, returns the error.

    test_loader -- DataLoader object with the test dataset.
    model -- Neural Network to be evaaluated.
    criterion -- Loss function.
    """

    model.eval()
    error = 0
    with torch.no_grad():
        for batch_idx, (x, y) in enumerate(test_loader):
            error += criterion(model(x), y).item()   # cummulative error/batch
    model.train()

    return error / batch_idx # ~ error over all testing samples
```

Results

We test the model on the [chart](https://finance.yahoo.com/quote/ANET/?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig=AQAAACcYE76pxfIMIu7x_2P-BSD9ufvEL1Gjz1_QKIR6uRffsIMUP_xzVzuOGJusIHMIbajTyvL-8LbgoetbBKZ1Cl6TfQToIJ5gV4x3DG9aOT3lALbP59gjIJZmmihz52uicQWfUuMDu5jnlMXgH4t3BXl_CduQBEgS-FzwuyFF_4vd) for [Arista Networks](https://www.arista.com/en/) (ANET) for the dates of 06/06/2014 to 08/18/2020. The train/test split follows an 80/20 split to mimic a production setting where we wish to predict prices in the future. Splitting the data randomly can lead to look-ahead bias.

```python
model = ANN(Layers=[in_features, 10, 10, out_features])
```

* Candles Plot of stock
* Plot of estimated values

Source code

* [Python script](https://github.com/Randomized-Neurais-Learners/FinML/blob/master/Forecasting/StocksANN.py)
* [Data request](https://github.com/Randomized-Neurais-Learners/FinML/blob/master/Data/getHistoricalDaily.py)
* [Data preparation](https://github.com/Randomized-Neurais-Learners/FinML/blob/master/Data/series_to_supervised.py)
* [FinML module](https://github.com/Randomized-Neurais-Learners/FinML/blob/master/FinML.py)


```python
from sympy import *
```

```python
p1 = Symbol("p1")
p2 = Symbol("p2")
p3 = Symbol("p3")
p4 = Symbol("p4")
p5 = 1-p1-p2-p3
p6 = 1-p1-p2-p4
lambd = Symbol("l")

y1= Symbol("y1")
y2= Symbol("y2")

expr = y1*y2*p1 + (1-y1)*(1-y2)*p2 + y1*(1-y2)*(lambd*p3 + (1-lambd)*p4) + (1-y1)*y2*(lambd*p4 + (1-lambd)*p5)
print(simplify(expand(expr)))
```

    -l*p1*y1*y2 + l*p1*y2 - l*p2*y1*y2 + l*p2*y2 - 2*l*p3*y1*y2 + l*p3*y1 + l*p3*y2 - l*p4*y1 + l*p4*y2 + l*y1*y2 - l*y2 + 2*p1*y1*y2 - p1*y2 + 2*p2*y1*y2 - p2*y1 - 2*p2*y2 + p2 + p3*y1*y2 - p3*y2 - p4*y1*y2 + p4*y1 - y1*y2 + y2

```python

```


# Basic Regression Analysis with Time Series Data
## The Nature of Time Series Data

One key feature that distinguishs time series data from cross-sectional data is, temporal ordering. And another one, that the OLS estimators are considered to be $r.v.$s while the time series data set, is considered as a ***realization*** of a **stochastic process**. And since time is irreversible we can only obtain one **realization**. The **sample size** for a time series data set is the *number of time periods* over which we observe the variables of interest.

## Examples of Time Series Regression Models
### Static Models

Suppose that we have time series data available on two variables, $y$ and $z$, where $y_t$ and $z_t$ are dated contemporaneously (marked with same date). Then, we have a typical ***static model*** relating $y$ to $z$:

$$\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\plim}{plim}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\bbspace}{\;\;\;\;\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\given}[1]{\left. #1 \right|}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\asim}{\overset{\text{a}}{\sim}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\II}{\mathbb{I}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\AcA}{\mathcal{A}}
\newcommand{\FcF}{\mathcal{F}}
\newcommand{\AsA}{\mathscr{A}}
\newcommand{\FsF}{\mathscr{F}}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathcal{N}\left( #1 \right)}
\newcommand{\Exp}[1]{\mathrm{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}
\newcommand{\Avar}[1]{\mathrm{Avar}\left[ #1 \right]}
\newcommand{\Cov}[1]{\mathrm{Cov}\left( #1 \right)}
\newcommand{\Corr}[1]{\mathrm{Corr}\left( #1 \right)}
\newcommand{\ExpH}{\mathrm{E}}
\newcommand{\VarH}{\mathrm{Var}}
\newcommand{\AVarH}{\mathrm{Avar}}
\newcommand{\CovH}{\mathrm{Cov}}
\newcommand{\CorrH}{\mathrm{Corr}}
\newcommand{\ow}{\text{otherwise}}y_t = \beta_0 + \beta_1 z_t + u_t,\bspace t = 1,2,\dots, n$$

Usually, the static model is postulated when a change in $z$ at time $t$ is believed to have an immediate effect on $y$: $\Delta y_t = \beta_1 \Delta z_t$, when $\Delta z_t = 0$. Another possible reason is we want to inspect the tradeoff between $y$ and $z$.

### Finite Distributed Lag Models
In a finite distributed lag (***FDL***) model, one or more variables with a lag can affect $y$. An **FDL** of order $k$ can be written as

$$y_t = \alpha_0 + \delta_0 z_t + \delta_1 z_{t-1} + \cdots + \delta_k z_{t-k} + u_t$$

Here, $\delta_0$ is called the ***impact propensity*** or ***impact multiplier***. And the ***lag distribution*** can be obtained by graphing the $\delta_j$ as a function of $j$, and we plot the graph uses estimated values.

Given a permanent increase in $z$, we find that $y$ also increases permanently by $\sum_{i=0}^k \delta_i$ multiply that increase in $z$, after at most $k$ periods. We call this permanent increase in $y$, the ***long-run propensity*** (***LRP***), or ***long-run multiplier***. The **LRP** is often of interest in distributed lag models.

We can transfer the **FDL** model to **static** model by setting the lagged variables to $0$. Sometimes, the primary purpose assuming this model is to test whether $z$ has a lagged effect on $y$. We also omit $z_t$, the **impact propensity** occasionally.

And for any time horizon $h$, we can define the ***cumulative effect*** as $\delta_0 + \delta_1 + \cdots + \delta_h$, as the change in the expected outcome $h$ periods after a permanent, one unit increase in $z$. And the estimated **cumulative effect** can be plotted as a function of $h$. Then **LRP** can be written as

$$\text{LRP} = \delta_0 + \delta_1 + \cdots + \delta_k$$

Multicollinearity in **FDL** of order $k$ happens a lot. Estimating individual $\delta_j$ precisely would be hard. $\text{LRP}$, however, can often be estimated nicely.

**e.f.1**

Suppose that $\text{int}_t = 1.6 + 0.48 \text{inf}_t - 0.15\text{inf}_{t-1} + 0.32\text{inf}_{t-2} + u_t$ where $\text{int}$ is an interest rate and $\text{inf}$ is the inflation rate. What are the impact and long-run propensities?

>$0.48$ and $0.48-0.15+0.32 = 0.65$
***
### A Convention about the Time Index
We can always start the time index at $t=1$. As for the initial observation, though the dependent variable at $t=1$ should be related to $z_{1}$, $z_0$ and $z_{-1}$. These three are the initial values in our sample.

## Finite Sample Properties of OLS under Classical Assumptions
### Unbiasedness of OLS

$Assumption$ $\text{TS}.1$

$\bspace$Linear in Parameters

>The stochastic process $\CB{\P{x_{t1},x_{t2}, \dots, x_{tk} ,y_t}: t = 1,2,\dots,n}$ follows the linear model:
>
>$$y_t = \beta_0 + \beta_1 x_{t1} + \cdots + \beta_k x_{tk} + u_t$$
>
>where $\CB{u_t:t=1,2,\dots,n}$ is the sequence of errors or disturbances. Here, $n$ is the number of observations (time periods).

Here, $x_{tj}$ denotes the observation at time period $t$, of $j^{\text{th}}$ **explanatory variables**, or **regressors**, or **independent variables**. And just like the terminology in cross-sectional regression, we call $y_t$ the **dependent variable**, **explained variable**, or **regressand**.

And we should think of $\text{TS}.1$ essentially, as same as $\text{MLR}.1$. Let $\mathbf{x}_t = \P{x_{t1}, x_{t2}, \dots,x_{tk}}$ denote the set of all independent variables in the equation at time $t$. Further let $\mathbf{X}$ denotes the collection of all independent variables for all time periods. $X$ can be thought as a matrix with $n$ rows and $k$ columns.

The $t^{\text{th}}$ row of $\mathbf{X}$ is $\mathbf{x}_t$, consisting of all independent variables for time period $t$.

$Assumption$ $\text{TS}.2$

$\bspace$No Perfect Collinearity

>In the sample (and therefore in the underlying time series process), no independent variable is constant nor a *perfect linear* combination of the others.

The issues are essentially same with cross-sectional data.

$Assumption$ $\text{TS}.3$

$\bspace$Zero Conditional Mean

>For each $t$, the expected value of the error $u_t$, given the explanatory variables for *all* time periods, is $0$. Mathematically,
>
>$$\Exp{u_t\mid \mathbf{X}} = 0,\bspace t = 1,2,\dots,n$$

And if $u_t$ is independent of $\mathbf{X}$ and $\Exp{u_t} = 0$, then this assumption automatically holds. One special condition of this assumption, are the ***contemporaneously exogenous*** explanatory variables. In conditional mean terms, we write

$$\Exp{u_t\mid x_{t1},x_{t2},\dots x_{tk}} = \Exp{u_t\mid \mathbf{x}_t} = 0 \Rightarrow \Corr{x_{tj},u_t} = 0,\forall\; j$$

The complete assumption $\text{TS}.3$ is called the ***strict exogeneity***, where $u_t$ must be uncorrelated with $x_{sj}$ where $s\neq t$.

$Remark$

>**Exogeneity** can gurantee the consistency while unbiasedness requires **strict exogeneity**.
>
>And another way to interpret $\text{TS}.3$ is that the average value of $u_t$ is unrelated to the independent variables in all time periods. It really doesn't matter whether there's correlation in the independent variables or in the $u_t$ across time.
***

How would $\text{TS}.3$ fails? Two leading candidates for failure are *omitted variables* and *measurement error* in some of the regressors. And to keep the **strict exogeneity** assumption in **static** regression model, we need to make sure that $u_t$ is uncorrelated with not just $z_t$, but also the past and future value of $z$. This implies that

- $z$ has no lagged effect on $y$, otherwise the model is wrong.
- Changes in error term $u_t$ today, will have NO chance to cause future changes in $z$. There's no feedback from $y$ to future values of $z$.

And in **finite distributed lag** model, the second implication keeps. Feedback from $u$ to future $z$ should be ruled out.

**e.f.2**

In the **FDL** model $y_t = \alpha_0 + \beta_0 z_t + \beta_1 z_{t-1} +u_t$, what to assume about the sequence $\CB{z_0,z_1,\dots,z_n}$ in order for $\text{TS}.3$ to hold?

>The explanatory variables are $x_{t1} = z_t$ and $x_{t2} = z_{t-1}$. The absence of perfect collinearity means that these CANNOT be constant, and there CANNOT be an exact linear relationship between them in the sample.
***
$Theorem.1$ Unbiasedness of OLS

>Under assumptions $\text{TS}.1$ through $\text{TS}.3$, the OLS estimators are unbiased, conditional on $\mathbf X$, and therefore unconditionally as well: $\Exp{\hat\beta_j} = \beta_j$, $j = 0,1,\dots,k$

### The Variances of the OLS Estimators and the Gauss-Markov Theorem

$Assumption$ $\text{TS}.4$

$\bspace$Homoskedasticity

>Conditional on $\mathbf X$, the variance of $u_t$ is the same for all $t$: $\Var{u_t\mid \mathbf X} = \Var{u_t} = \sigma^2$, for $t=1,2,\dots,n$

And if not, we say that the errors are ***heteroskedastic***. Homoskedasticity says two things

- $u_t$ and $\mathbf{X}$ are independent
- $\Var{u_t}$ is a constant over time

$Assumption$ $\text{TS}.5$

$\bspace$No serial correlation

>Conditional on $\mathbf{X}$, the errors in two different time periods are uncorrelated: $\Corr{u_t,u_s\mid \mathbf{X}} = 0$, $\forall t\neq s$

And currently we'll only consider this assumption as $\Corr{u_t,u_s} = 0$, $\forall t\neq s$. When it's failed, we say the errors suffer from ***serial correlation***, or ***autocorrelation***. And also note that this assumption rules out the temporal correlation only in the error term, not the independent variables.

You may ask why this is not assumed in cross-sectional observations. That's because the random sampling assumption, which guarantee that $u_i$ and $u_h$ are independent for any two observations $i$ and $h$. And actually if random sampling assumption is violated, this could be an ideal substitute.

$\text{TS}.1$ through $\text{TS}.5$ are the **Gauss-Markov assumptions** for time series applications.

$Theorem.2$ OLS Sampling Variances

> Under the time series Gauss-Markov Assumptions $\text{TS}.1$ through $\text{TS}.5$, the variance of $\hat\beta_j$, conditional on $\mathbf{X}$, is
>
>$$\Var{\hat\beta_j\mid \mathbf X} = \ffrac{\sigma^2}{\text{SST}_j \P{1-R_j^2}},\bspace j=1,2,\dots,k$$
>
>where $\text{SST}_j$ is the total sum of squares of $x_{tj}$ and $R_j^2$ is the $R$-squared from the regression of $x_j$ on other independent variables.

This theorem and its proof are just like thoes in cross-sectional case.

$Theorem.3$ Unbiased estimation of $\sigma^2$

> Under $\text{TS}.1$ through $\text{TS}.5$, the estimator $\hat\sigma^2 = \ffrac{\text{SSR}}{df}$ is an unbiased estimator of $\sigma^2$, where $df = n-k-1$

$Theorem.4$ Gauss-Markov Theorem

>Under $\text{TS}.1$ through $\text{TS}.5$, the OLS estimators are the best linear unbiased estimators conditional on $\mathbf X$.

**e.f.3**

In the **FDL** model $y_t = \alpha_0 + \delta_0 z_t + \delta_1 x_{t-1} + u_t$, explain the multicollinearity in the explanatory variables.

>Suppose that $\CB{z_t}$ moves slowly over time, then $z_t$ and $z_{t-1}$ can be highly correlated, as can be seen in logs of many economic time series.

### Inference under the Classical Linear Model Assumptions

$Assumption$ $\text{TS}.6$

$\bspace$Normality

> The errors $u_t$ are independent of $X$ and are $i.i.d.$ of $\N{0,\sigma^2}$

Note that this assumption implies $\text{TS}.3$ through $\text{TS}.5$. And taking all assumptions together, we have

$Theorem.5$ Normal sampling distributions

>Under $\text{TS}.1$ through $\text{TS}.6$, the $\text{CLM}$ assumptions for time series, the OLS estimators are **normally distributed**, *conditional* on $\mathbf X$. Further, under the null hypothesis, each $t$ statistic has a $t$ distribution, and each $F$ statistic has an $F$ distribution. The usual construction of confidence intervals is also valid.

## Functional Form, Dummy Variables, and Index Numbers
We use natural logarithm often, that shows time series regression with constant percentage effects, in **static** or **lag** models.

And in **FDL**, **impact multiplier** $\delta_0$ is also called the ***short-run elasticity***, which is the immediate percentage change in $y_t$ given a $1\%$ increase in $z_t$. And similarly, **long-run multiplier** changes into ***long-run elasticity***.

Dummy variables is also very useful since it can indicate whether certain event happened at time $t$. They are key to ***event study***. And one more thing to notice is the ***index number***, which is usually called index. Like S&P index, they aggregate a vast amount of information into a single quantity, comparing a predetermined ***base period*** and the ***base value*** at that time.

## Trends and Seasonality
### Characterizing Trending Time Series

One popular formulation with trending behavior is to write the series $\CB{y_i}$ as

$$y_t = \alpha_0 + \alpha_1 t + e_t,\bspace t=1,2,\dots$$

where, in the simplest case, $\CB{e_t}$ is an $i.i.d.$ sequence with $\Exp{e_t} = 0$ and $\Var{e_t} = \sigma_t^2$ and we call this ***linear time trend***. We'll interpret $\alpha_1$ as, *holding all other factors fixed, the change in $y_i$ from one period to the next due to the passage of time*.

This **linear time trend** implies that if $\Delta e_t = 0$, $\Delta y_t = y_t - y_{t-1} = \alpha1 + \Delta e_t = \alpha_1$. And another way to think this is that *the average value of this sequence is a linear function of time*: $\Exp{y_t} = \alpha_0 + \alpha_1 t$; and as for the variance we have $\Var{y_t} = \Var{e_t} = \sigma_e^2$.

**e.f.4**

Can a **linear trend** with $\alpha_1<0$ be realistic for all future time periods?

>It depends. It'll be more and more negative as $t$ gets larger. For some $y_t$ which can never be negative, a linear time trend with a negative trend coefficient cannot represent it in all future time periods.
***
Another typical trend is the ***exponential trend*** where we would write

$$\log\P{y_t} = \beta_0 + \beta_1 t + e_t,\bspace t =1,2,\dots$$

To interpret first we notive the approximation for small changes

$$\Delta \log\P{y_t} = \log\P{y_t} - \log\P{y_{t-1}} \approx\ffrac{y_t - y_{t-1}}{y_{t-1}}$$

The right hand side is called the ***growth rate*** in $y$ from period $t-1$ to period $t$. And if further assumed that $\Delta e_t = 0$, we have $\forall\,t$, $\Delta \log\P{y_t} = \beta_1$.

### Using Trending Variables in Regression Analysis

***Spurious regression problem*** would happen if there's a relationship between two or more trending variables. And to eliminate this problem we can simply add the time trend. And commonly, adding trends $t$ and $t^2$ would be enough.

$Remark$

>We'll include a trend when
>
>- if the dependent variable displays an obvious trending behaviour
>- if both the dependent and some independent variables have trends
>- if only some of the independent variables have trends, and *only when a trend is added* their effect on the dependent variable will be visible

### A Detrending Interpretation of Regressions with a Time Trend
Including a time trend is like detrending the original data series before regression. For instance we first obtain the fitted equation

$$\hat y_t = \hat \beta_0 + \hat \beta_1 x_{t1}+ \hat\beta_2 x_{t2} + \hat\beta_3 t$$

Similar to what we did in Chap_03, we can apply three-step OLS regression. $x_{t2}$ on $t$, then $x_{t2}$ on $x_{t1}$ then $y$ on $r_{t2}$.

$\P{1}$ Regress each of $y_t$, $x_{t1}$ and $x_{t2}$ on a constant and the time trend $t$ and save the residuals as $\ddot y_t$, $\ddot x_{t1}$ and $\ddot x_{t2}$.

$$
\begin{align}
y_t &= \hat\alpha_0 + \hat\alpha_1 t + \ddot y_t \\
x_{t1} &= \hat\eta_0 + \hat\eta_1 t +\ddot x_{t1}\\
x_{t2} &= \hat\xi_0 + \hat\xi_1 t + \ddot x_{t2}
\end{align}$$

Then $\ddot y_t$, $\ddot x_{t1}$ and $\ddot x_{t2}$ can be regarded as the detrended variables.

$\P 2$ Run the regression of $\ddot y$ on $\ddot x_{t1}$ and $\ddot x_{t2}$. This regression exactly yields $\hat\beta_1$ and $\hat\beta_2$. Here's a proof for one-variable case.

$Proof$ case: $k=1$

>Write the model as $y_t = \beta_0 + \beta_1 x_{t1} + \beta_2 t + u_t$. To detrend, first we have $y_t = \hat\alpha_0 + \hat\alpha_1 t + \ddot y_t$ and $x_{t1} = \hat\eta_0 + \hat\eta_1 t + \ddot x_{t1}$. Then regress the equation
>
>$$\ddot y_t = \hat\theta_1 \ddot x_{t1} + v_t \Rightarrow \hat\theta_1 = \ffrac{\sum\limits_{t=1}^{n} \ddot x_{t1} \ddot y_t}{\sum\limits_{t=1}^{n} \ddot x_{t1}^2}$$
>
>Then using the two-step OLS estimation, we have $x_{t1} = \hat\delta_0 + \hat\delta_1 t + \hat r_{t1}$ and $y_t = \hat\beta_0 + \hat\beta_1 \hat r_{t1} + w_t$. Then,
>
>$$\hat\beta_1 = \ffrac{\sum\limits_{t=1}^n \hat r_{t1} y_t}{\sum\limits_{t=1}^n \hat r_{t1}^2}$$
>
>Remember that $\sum\limits_{t=1}^n t\cdot \hat r_{t1} = 0$ and notice that $\hat r_{t1} = \ddot x_{t1}$, then
>
>$$\hat\beta_1 = \ffrac{\sum\limits_{t=1}^n \hat r_{t1} \P{\hat\alpha_0 + \hat\alpha_1 t + \ddot y_t}}{\sum\limits_{t=1}^n \hat r_{t1}^2} = \ffrac{\sum\limits_{t=1}^n \ddot x_{t1}\ddot y_t}{\sum\limits_{t=1}^n \ddot x_{t1}^2} + \alpha_0 \ffrac{\sum\limits_{t=1}^n \hat r_{t1}}{\sum\limits_{t=1}^n \hat r_{t1}^2} + \alpha_1 \ffrac{\sum\limits_{t=1}^n \hat r_{t1}  \cdot t }{\sum\limits_{t=1}^n \hat r_{t1}^2} = \hat\theta_1$$

So we may draw the conclusion that *the OLS coefficients in a regression **including a trend** are the **same** as the coefficients in a regression **without a trend** but where **all** the variables have been **detrended before the regression***.

### Computing $R$-Squared when the Dependent Variable is Trending
Usually it'll be higher, comparing with typical $R$-squareds for cross-sectional data, expecially when the dependent variable is trending.

The adjusted $R$-squared is written as $\bar R^2 = 1 - \ffrac{\hat\sigma_u^2}{\hat\sigma_y^2}$, where $\hat \sigma_u^2$ is the **unbiased estimator of the error variance**, $\hat\sigma_y^2 = \ffrac{\text{SST}}{n-1} = \ffrac{\sum_{t=1}^n \P{y_t - \bar y}^2}{n-1}$. With these formulas we can easily calculate the **error variance** *given that a time trend is included in the regression.* However, $\ffrac{\text{SST}}{n-}$ may overestimate the variance in $y_t$, because it does NOT account for the trend in $y_t$.

And if there's any polynomial trend, we can calculate the $R$-squared of $y = \beta_0 + \beta_1 x_{t1} + \beta_2 x_{t2} + \beta_3 t + u_t$ by

1. detrend the dependent variable: regress $y_t$ on $t$ to obtain the residuals $\ddot y_t$
2. regress the residual on the independent variables: $\ddot y_t$ on $x_{t1}$, $x_{t2}$ and $t$
3. $R^2 = 1 - \ffrac{\text{SSR}}{\sum_{t=1}^n \ddot y_t^2}$ where $\text{SSR}$ is identical to the sum of squared residuals from the original model

We prefer using this because it's smaller and has net out the effect of the time trend, based on the fact that $\sum_{t=1}^n \ddot y_t^2 \leq \sum_{t=1}^n \P{y_t - \bar y}^2$. And the corresponding adjusted $R$-squared is $\bar R^2 = 1 - \ffrac{\frac{1}{n-4}\text{SSR}}{\frac{1}{n-2}\sum_{t=1}^n \ddot y_t^2}$ where $n-4$ is the $df$ of the original model and $n-2$ is the $df$ of the restricted model.

***

One more thing, in computing the $R$-squared form of an $F$ statistic for testing multiple hypotheses, usual $R$-squared without any detrending would be enough.

### Seasonality

Some data series display seasonal patterns, and are often ***seasonally adjusted*** before they are reported for public use, meaning the seasonal factors are removed. However there's time when we work with seasonally unadjusted data and that's the time we'll add a set of ***seasonal dummy variables*** to account for seasonality in the dependent variable, the independent variables, or both. We call this ***deseasonalizing*** the data and we'll write

$$y_t = \beta_0 + \beta_1 x_{t1} + \beta_2 x_{t2} + \cdots + \beta_k x_{tk} + u_t\\
\Downarrow\\
y_t = \beta_0 + \beta_1 x_{t1} + \beta_2 x_{t2} + \cdots + \beta_k x_{tk} + u_t + \delta_1 \text{Jan}_t + \delta_2 \text{Feb}_t + \cdots + \delta_{11} \text{Nov}_t$$

to eliminate the monthly effect to the model. Note that here we would put only $11$ months rather than $12$ to avoid the... some kind of trap, with $\text{Dec}$ as the base month, $\beta_0$ is the intercept for December. Besides, thougth $F$ test, $\delta_1 = \cdots = \delta_{11} = 0$, we can determine whether there's seasonality in $y_t$ or not.

Further discussions are skipped.
***

**e.f.5**

What's the intercept for March? And why the seasonal dummy variables satisfy the strict exogeneity assumption?

>For March, its intercept is $\mathbf \beta_0 + \delta_3$. Seasonal dummy variables are strictly exogenous because they follow a deterministic pattern. For example, the months do not change based upon whether either the explanatory variables or the dependent variable change.
***

FinMath/Econometrics/Chap_10_Basic Regression Analysis with Time Series Data.ipynb


# Integrals

**This assignment is due Friday, February 21 at 5PM by email.**

For this assignment you will be computing the overlap, kinetic, and dipole integrals.  You will then incorporate your integral code into your SCF code from the previous programming assignment. Finally, using your converged one-particle density matrix you are to compute the dipole moment of the molecule.

## Recurrence formula for one-electron integrals over Gaussian functions
### Cartesian Gaussian functions
We write the unnormalized Cartesian Gaussian function with origin at $\mathbf R$ as
\begin{equation}
\phi(\mathbf r; \alpha, \mathbf n, \mathbf R) = (x - \mathbf R_x)^{n_x} (y - \mathbf R_y)^{n_y} (z - \mathbf R_z)^{n_z} \exp[-\alpha (\mathbf r - \mathbf R)^2
\end{equation}
where $\mathbf r = (x, y, z)$ represents the coordinates of the electron, $\alpha$ is the orbital exponent, and $\mathbf n$ denotes a set of nonnegative integers $n_x$, $n_y$, and $n_z$:
\begin{equation}
\mathbf n = (n_x, n_y, n_z) \ .
\end{equation}
The normalization constant of the function is then given by
\begin{equation}
N(\alpha, \mathbf n) = \left(\frac{2\alpha}{\pi}\right)^{3/4} (4\alpha)^{(n_x + n_y + n_z)/2} [(2n_x-1)!!\ (2n_y-1)!!\ (2n_z-1)!!]^{-1/2}
\end{equation}

Letting $\lambda$ be defined as the sum of $n_x$, $n_y$, and $n_z$, one notes that it is closely related to the total angular momentum quantum number. $\lambda$ and $\mathbf n$ will be hereafter termed the angular momentum and the angular momentum index, respectively. The functions with $\lambda$ equal to $0,1,2,\ldots$, are referred to as $s,p,d,\ldots$, respectively. A set of $(\lambda+1)(\lambda+2)/2$ functions at $\mathbf R$ associated with the same angular momentum $\lambda$ and orbital exponent $\alpha$ constitute a *shell*, and the functions in the shell are *components* of the shell. The single component of the *s* shell with the angular momentum index $\mathbf 0 = (0,0,0)$ is usually designated *s*. The components of the *p* shell have the angular momentum indices $\mathbf 1_i\ (i=x,y,z)$, where  $\mathbf 1_i$ is defined by
\begin{equation}
\mathbf 1_i = (\delta_{ix},\delta_{iy},\delta_{iz})
\end{equation}
with Kronecker's deltas. These components will be designated $p_x$, $p_y$, and $p_z$. The components of the  *d* shell with the angular momentum indices $\mathbf 1_i + \mathbf 1_j\ (i,j=x,y,z)$ will be designated $d_{x^2}$, $d_{y^2}$, $d_{z^2}$, $d_{xy}$, $d_{xz}$, and $d_{yz}$. In later discussions all the components  in a shell are assumed to be exhausted as the basic functions, and to be treated together in the computation of molecular integrals.

First we give the basic equations for Cartesians Gaussian functions that will be utilized in the following derivations of molecular integrals. The Cartesian Gaussian functions satisfy the differential relation
\begin{equation}
\frac{\partial}{\partial R_i} \phi(\mathbf r; \alpha, \mathbf n, \mathbf R) = 2\alpha \phi(\mathbf r; \alpha, \mathbf n + \mathbf 1_i, \mathbf R) - N_i(\mathbf n)\phi(\mathbf r; \alpha, \mathbf n - \mathbf 1_i, \mathbf R) \qquad (i=x,y,z)
\end{equation}
where $N_i(\mathbf n)$, standing for $n_i$, is meant to take the value of the *i* component of the angular momentum index $\mathbf n$, and thus  $N_i(\mathbf 1_j)$ plays the same role as the Kronecker's delta $\delta_{ij}$.

### Two-center overlap integrals
Two-center overlap integrals over unnormalized Cartesian Gaussian functions are of the form:
\begin{equation}
(\mathbf a|\mathbf b) = \int d\mathbf r\ \phi(\mathbf r; \alpha_a, \mathbf a, \mathbf A)\phi(\mathbf r; \alpha_b, \mathbf b, \mathbf B)
\end{equation}
According to the differential relation above the integral $(\mathbf a+\mathbf 1_i|\mathbf b)$ can be decomposed as
\begin{equation}
(\mathbf a+\mathbf 1_i|\mathbf b) = \frac{1}{2\alpha_a} \frac{\partial}{\partial \mathbf A_i} (\mathbf a|\mathbf b) + \frac{1}{2\alpha_a} N_i(\mathbf a) (\mathbf a - \mathbf 1_i|\mathbf b)
\end{equation}
Here the integral can be $(\mathbf a | \mathbf b)$ can be factored as
\begin{equation}
(\mathbf a | \mathbf b) = \kappa_{ab}\ I_x(n_{ax},n_{bx})\ I_y(n_{ay},n_{by})\ I_z(n_{az},n_{bz})
\end{equation}
where
\begin{align}
\kappa_{ab} =& \exp[-\xi (\mathbf A - \mathbf B)^2] \\
\xi =& \frac{\alpha_a \alpha_b}{\alpha_a + \alpha_b} \\
\alpha =& \alpha_a + \alpha_b \\
\mathbf P =& \frac{\alpha_a \mathbf A + \alpha_b \mathbf B}{\alpha_a + \alpha_b}
\end{align}
and
\begin{align}
I_i(n_{ai}, n_{bi}) =& \left(\frac{\pi}{\alpha_a + \alpha_b}\right)^{\tfrac{1}{2}} \\
 &\times \sum_{k_{ai}=0}^{n_{ai}} \sum_{k_{bi}=0}^{n_{bi}} \binom{n_{ai}}{k_{ai}} \binom{n_{bi}}{k_{bi}} \\
 &\times (\mathbf P_i - \mathbf A_i)^{n_{ai}-k_{ai}} (\mathbf P_i - \mathbf B_i)^{n_{bi}-k_{bi}} \\
 &\times \frac{(k_{ai} + k_{bi} - 1)!!}{(2\alpha)^{k_{ai}+k_{bi}}}
\end{align}
Differentiating $\kappa_{ab}$ and $I_i(n_{ai}, n_{bi})$ with respect to $A_i$, we have
\begin{equation}
\frac{1}{2\alpha_a}\ \frac{\partial}{\partial A_i} \kappa_{ab} = (P_i - A_i) \kappa_{ab}
\end{equation}
and
\begin{equation}
\frac{1}{2\alpha_a}\ \frac{\partial}{\partial A_i} I_i(n_{ai}, n_{bi}) = n_{ai}\left\{\frac{1}{2\alpha} - \frac{1}{2\alpha_a}\right\} I_i(n_{ai} - 1, n_{bi}) + n_{bi} \frac{1}{2\alpha} I_i(n_{ai}, n_{bi} - 1)
\end{equation}
Substituting these two equations into the either expression for $(\mathbf a+\mathbf 1_i|\mathbf b)$ yields the following recursion relations. **The following three equations are what you need to program:**
\begin{equation}
(\mathbf a+\mathbf 1_i|\mathbf b) = (P_i - A_i)(\mathbf a|\mathbf b) + \frac{1}{2\alpha} N_i(\mathbf a)(\mathbf a-\mathbf 1_i|\mathbf b) + \frac{1}{2\alpha} N_i(\mathbf b)(\mathbf a|\mathbf b-\mathbf 1_i)
\end{equation}
and similarily
\begin{equation}
(\mathbf a|\mathbf b+\mathbf 1_i) = (P_i - B_i)(\mathbf a|\mathbf b) + \frac{1}{2\alpha} N_i(\mathbf a)(\mathbf a-\mathbf 1_i|\mathbf b) + \frac{1}{2\alpha} N_i(\mathbf b)(\mathbf a|\mathbf b-\mathbf 1_i)
\end{equation}
The integral over $s$ functions is given by
\begin{equation}
(\mathbf 0_A | \mathbf 0_B) = \left(\frac{\pi}{\alpha}\right)^{3/2} \exp[-\xi(\mathbf A-\mathbf B)^2]
\end{equation}

Given the recursive equations given above it should seen that the recursion only depends on $\mathbf P-\mathbf A$, $\mathbf P-\mathbf B$, $\alpha$, and the target angular momentum of $\mathbf a$ and $\mathbf b$. The starting point of the recursion $(\mathbf 0_A | \mathbf 0_B)$ can be accounted for after the recursion has been performed.

### Recurrence expressions for overlap integrals over $s$, $p$, and $d$  Cartesian Gaussian funcions
To serve as an example of the recurrence relations here are the explicit equations for several cases worked out. This may aid you in the implementation of your recursion function.
\begin{align}
(s|s) =& \left(\frac{\pi}{\alpha}\right)^{3/2} \exp\left\{-\xi(\mathbf A-\mathbf B)^2\right\} \\
(p_i|s) =& (P_i - A_i)(s|s) \\
(p_i|p_j) =& (P_j - B_j)(p_i|s) + \frac{\delta_{ij}}{2\alpha}(s|s) \\
(d_{ij}|s) =& (P_j-A_j)(p_i|s) + \frac{\delta_{ij}}{2\alpha}(s|s) \\
(d_{ij}|p_k) =& (P_k-B_k)(d_{ij}|s) + \frac{\delta_{ik}}{2\alpha}(p_j|s) + \frac{\delta_{jk}}{2\alpha}(p_i|s) \\
(d_{ij}|d_{kl}) =& (P_l-B_l)(d_{ij}|p_k) + \frac{\delta_{il}}{2\alpha}(p_j|p_k) + \frac{\delta_{jl}}{2\alpha}(p_i|p_k) + \frac{\delta_{kl}}{2\alpha}(d_{ij}|s) \\
\\
& (i,j,k,l = x,y,z)
\end{align}

### Cartesian function ordering that Psi4 uses:
\begin{align}
\mathbf s &= s \\
\mathbf p &= p_x, p_y, p_z \\
\mathbf d &= d_{x^2}, d_{xy}, d_{xz}, d_{y^2}, d_{yz}, d_{z^2} \\
\mathbf f &= f_{x^3}, f_{x^2y}, f_{x^2z}, f_{xy^2}, f_{xyz}, f_{xz^2}, f_{y^3}, f_{y^2z}, f_{yz^2}, f_{z^3}
\end{align}

## One electron integrals
It is now time to proceed to computing the following one-electron integrals:
- overlap
- kinetic
- dipole moment

I am not asking you to compute the nuclear attraction integrals or the two-electron repulsion integrals because of their added complexity.

### Kinetic energy integrals
Before proceeding to an analysis of the kinetic energy integral, it will prove convenient to establish a short hard notation for integrals realted to the overlap integral:
\begin{equation}
(0|0) = \int G(\alpha_1, \mathbf A, l_1, m_1, n_1) G(\alpha_2, \mathbf B, l_2, m_2, n_2)\ d\tau
\end{equation}
The symbol $(+\mathrm{n}|0)_{\mathrm{x}}$ will denote an *overlap integral* of the form given by the above equation, except that the quantum number $l_1$ has been incremented by n. Similar notations will apply to the $m_1$ and $n_1$ quantum numbers by the use of subscripts x and y.  We will also use the symbols $(0|+\mathrm{n})_{\mathrm{x}}$, where we have incremented the quantum number $l_2$ by n. You should see the pattern.

The kinetic energy integral is defined as
\begin{equation}
T_{ij} = -\frac{1}{2} \int \phi_i(1) \nabla_1^2 \phi_j(1)\ d\tau_1
\end{equation}
and as
\begin{equation}
KE = - \frac{1}{2}  \int G(\alpha_1, \mathbf A, l_1, m_1, n_1) \left(\frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y^2} + \frac{\partial^2}{\partial z^2}\right) G(\alpha_2, \mathbf B, l_2, m_2, n_2)\ d\tau
\end{equation}

Evaluate the kinetic energy integral for $I_x$, $I_y$, and $I_z$ and add them up to obtain $T_{ij}$. The equations that you obtain are "unsymmetric" because the integrals which appear in the equations are of the form where the quantum numbers f the GTF centered on $\mathbf B$ are altered while those of the GTF on $\mathbf A$ are not.

Here is the symmetric form of the kinetic energy integral $I_x$ component.
\begin{equation}
I_x = \frac{1}{2}\Big\{l_1 l_2 (-1|-1)_{\mathrm{x}} + 4\alpha_1 \alpha_2 (+1|+1)_{\mathrm{x}} - 2\alpha_1 l_2 (+1|-1)_{\mathrm{x}} - 2\alpha_2 l_1 (-1|+1)_{\mathrm{x}}\Big\}
\end{equation}

### Dipole moment integrals
We discuss these non-energy integrals here because of the frequency of their use, and their close connection with the overlap and kinetic energy integrals. The dipole moment is defined with respect to a point in space $\mathbf C$. Almost always $\mathbf C$ is taken to be the center of mass. Fortunately, for you Psi4 automatically moves the molecule to be centered on the center of mass and thus $\mathbf C$ is $\mathbf 0$. The dipole moment integral is written for the $x$ direction as
\begin{equation}
d_{\mathrm{x}} =  \int G(\alpha_1, \mathbf A, l_1, m_1, n_1) G(\alpha_2, \mathbf B, l_2, m_2, n_2) \mathrm{x_c}\ d\tau
\end{equation}
and similarly for the operators $\mathrm{y_c}$ and $\mathrm{z_c}$. A convenient procedure is to redefine $\mathrm{x_c}$ in terms of  $\mathrm{x_A}$ or $\mathrm{x_B}$, and we will use $\mathrm{x_A}$. Thus
\begin{equation}
\mathrm{x_c} = \mathrm{x} - \mathrm{C_x} = (\mathrm{x - A_x}) + (\mathrm{A_x - C_x}) = \mathrm{x_A + \mathbf {AC}_x}
\end{equation}
and we find
\begin{equation}
d_\mathrm{x} = (+1|0)_{\mathrm{x}} + \mathbf{AC}_{\mathrm{x}} (0|0)
\end{equation}
Similarily for $d_{\mathrm{y}}$ and $d_{\mathrm{z}}$.

#### The x component
\begin{align}
\mu_x =& - \mu_{\mathrm{elec}} + \mu_{\mathrm{nuclear}}\\
\mu_x =& - \sum_{\mu \nu}^{\mathrm{AO}} D_{\mu \nu} d_{\mu \nu}^{x} + \sum_A^N Z_A X_A
\end{align}

You can obtain the nuclear contribution to the dipole $\mu_{\mathrm{nuclear}}$ from Psi4 using

    nucdip = molecule.nuclear_dipole()

#### The total dipole moment
\begin{equation}
\mu = \sqrt{\mu_x^2 + \mu_y^2 + \mu_z^2}
\end{equation}

## Load Psi4 and NumPy

```python
import psi4
import numpy as np
import pprint
pp = pprint.PrettyPrinter(indent=4)

np.set_printoptions(precision=10, linewidth=140, suppress=True)

# I'll show some plots
import matplotlib.pyplot as plt

# Load in the required options
from input import Settings

pp.pprint(Settings)

molecule = psi4.geometry(Settings['molecule'])

# For your integrals code to match Psi4's ordering you must include "puream=0"
basis = psi4.core.BasisSet.build(molecule, 'BASIS', Settings['basis'], puream=0)
mints = psi4.core.MintsHelper(basis)
```

## Determine some information about the basis set

```python
# Look at the help documentation for the basis set
help(basis)
help(psi4.core.GaussianShell)
```

```python
print('The number of shells: %d' % (basis.nshell()))
print('The number of basis functions: %d' % (basis.nao()))
```

```python
# Print out some information about the shells in the basis set
for ishell in range(basis.nshell()):
    shell = basis.shell(ishell)
    print('Shell %d:' % (ishell))
    print('  Atom: %d' % (shell.ncenter))
    print('  AM: %d' % (shell.am))
    print('  # Cartesian functions: %d' % (shell.ncartesian))
    print('  # of primitive Gaussians: %d ' % (shell.nprimitive))
    print('  function_index: %d' % (shell.function_index))
    print('  center: %f, %f, %f' % (molecule.x(shell.ncenter), molecule.y(shell.ncenter), molecule.z(shell.ncenter)))
```

### Let's plot some of the functions to see what they look like

```python
# This is to show you have to access individual shells and access their coefficients and exponents.
shell0 = basis.shell(0)

# generate a list of points to evaluate the basis function (R=0 to 10 in 100 points)
R = np.linspace(0, 2, 100, endpoint=True)

# Plot each primitive
for idx in range(shell0.nprimitive):
    v = shell0.coef(idx) * np.exp(-shell0.exp(idx) * R**2)
    plt.plot(R, v)
plt.show()
```

### Write the recursion function
Write a function that performs the OS recursion for a given set of $\mathbf{PA}$, $\mathbf{PB}$, $\alpha$, and two angular momenta (for $\mathbf a$ and $\mathbf b$) values. At this point do not worry about solving $(\mathbf 0 | \mathbf 0)$ just set it to $1.0$ and you'll account for it later.  This function should also return three matrices containing the x, y, and z results.

```python
import numpy as np
from collections import namedtuple
RecursionResults = namedtuple('RecursionResults', ['x', 'y', 'z'])

def os_recursion(PA, PB, alpha, AMa, AMb):
    if len(PA) != 3 or len(PB) != 3:
        raise "Coordinates must only be in terms of x, y, and z."

    # Allocate the x, y, and z matrices.
    #  Why do I add 1 here?
    x = np.zeros((AMa+1, AMb+1))
    y = np.zeros((AMa+1, AMb+1))
    z = np.zeros((AMa+1, AMb+1))

    # Perform the recursion

    # Return the results
    return RecursionResults(x, y, z)

results = os_recursion([0.0, 0.0, 0.0], [0.0, 0.0, 0.0], 1.0, 1, 1)
print(results)
print(results.x)
```

### Sample x, y, and z matrices

```python
import OS
```

```python
dipole = [x.to_array() for x in mints.ao_dipole()]
for i in dipole:
    print(i)

nucdip = molecule.nuclear_dipole()
print(nucdip[0], nucdip[1], nucdip[2])
```

```python
(x, y, z) = OS.os_recursion([ 0.,      -0.92825,  0.71873],[ 0.,       0.56594, -0.4382 ], 1.00430273, 4, 4)
print('x=\n', x)
print('y=\n', y)
print('z=\n', z)
```

    x=
     [[1.           0.           0.4978578521 0.           0.7435873226]
     [0.           0.4978578521 0.           0.7435873226 0.          ]
     [0.4978578521 0.           0.7435873226 0.           1.8510039364]
     [0.           0.7435873226 0.           1.8510039364 0.          ]
     [0.7435873226 0.           1.8510039364 0.           6.450757908 ]]
    y=
     [[ 1.            0.56594       0.8181459357  1.0265368565  1.8029194032]
     [-0.92825      -0.0274759529 -0.1959286192  0.2690782977  0.3707178017]
     [ 1.3595059146 -0.1548743251  0.5618328812 -0.0313362999  1.0893297555]
     [-2.1862344676  0.793244549  -0.8708213726  1.1361521414 -0.7044449004]
     [ 4.0598942281 -2.0560794453  2.4373247282 -2.4020720686  4.5434641546]]
    z=
     [[ 1.           -0.4382        0.6898770921 -0.7386267633  1.3540484295]
     [ 0.71873       0.1829103661  0.0595127408  0.4995089682 -0.4977293078]
     [ 1.014430665   0.2711272307  0.5683610434  0.0801676028  1.3111285054]
     [ 1.4447524999  0.8820362704  0.7377215448  1.4038768101  0.6063987846]
     [ 2.5535137801  1.7581757671  2.2573689969  2.2305859825  5.1898282359]]

```python
(x, y, z) = OS.os_recursion([ 1, 2,  3],[ 5, 6, 7], 3.14159, 4, 4)
print('x=\n', x)
print('y=\n', y)
print('z=\n', z)
```

    x=
     [[   1.              5.             25.1591550775  127.3873261629  648.9492526448]
     [   1.              5.1591550775   26.7507058528  139.3999479933  730.0466117291]
     [   1.1591550775    6.1140855427   32.3971245822  172.4468197559  922.0750184623]
     [   1.4774652326    7.9407824115   42.858321427   232.2877394388 1264.2394165031]
     [   2.0309214813   11.0951917808   60.854453731   335.0884587479 1852.3772723567]]
    y=
     [[    1.               6.              36.1591550775   218.8647913954  1330.4534877614]
     [    2.              12.1591550775    74.2281710853   454.9943221796  2800.2407468904]
     [    4.1591550775    25.5915507752   158.0816378383   980.2634581719  6101.8885482469]
     [    8.9549304651    55.7154347373   347.9369068189  2180.8347155592 13719.1777895295]
     [   19.8957128767   125.0751678698   789.0870882836  4995.8389272927 31740.1588867753]]
    z=
     [[     1.                7.               49.1591550775    346.342256628    2447.8675838083]
     [     3.               21.1591550775    149.7056363179   1062.4985572962   7564.0912662394]
     [     9.1591550775     65.0690160078    463.6760120246   3314.0970370544  23758.272514813 ]
     [    28.4323956977    203.3999479933   1459.3929889561  10501.8843665954  75792.366091623 ]
     [    89.6703652024    645.7931969829   4664.312410771   33784.8286252416 245406.5602687671]]

```python

```


# Application: Cavity Flow

One of the most common validation cases in CFD is the lid-driven cavity flow.  We take a square cavity filled with a fluid and set the velocity of the lid to some constant value.  The flow within the cavity is driven by the lid, a spiral flow pattern develops and two distinctive pressure zones are visible in the upper corners against the lid.

```python
import numpy
```

The Poisson equation is an elliptic PDE which almost always means using an iterative solver.  We're going to use the Jacobi method.  There are better ways, but that's beside the point.

Here's the pressure Poisson equation:

\begin{align}
p_{i,j}^{n+1} &= \frac{1}{4}\left(p_{i+1,j}^{n}+p_{i-1,j}^{n}+p_{i,j+1}^{n}+p_{i,j-1}^{n}\right) \\
&-\frac{\rho \Delta x}{16} \left( \frac{2}{\Delta t} \left(u_{i+1,j} - u_{i-1,j} + v_{i,j+1} - v_{i,j-1}\right) \right . \\
&-\frac{2}{\Delta x}\left(u_{i,j+1} - u_{i,j-1} \right) \left(v_{i+1,j} - v_{i-1,j} \right) \\
&- \left . \frac{\left(u_{i+1,j} - u_{i-1,j} \right)^2}{\Delta x}
- \frac{ \left(v_{i,j+1} - v_{i,j-1} \right)^2 }{\Delta x} \right) \\
\end{align}

That looks a little nasty, but we only care about the top line when we iterate, since the bottom three lines depend only on values that don't change when we're correcting the pressure field.  Because it doesn't change, we break it out into a separate function.

```python
def velocity_term(b, rho, dt, u, v, dx):
    b[1:-1, 1:-1] = (
        rho * dx / 16 *
        (2 / dt * (u[2:, 1:-1] -
                   u[:-2, 1:-1] +
                   v[1:-1, 2:] -
                   v[1:-1, :-2]) -
        2 / dx * (u[1:-1, 2:] - u[1:-1, :-2]) *
                 (v[2:, 1:-1] - v[:-2, 1:-1]) -
        (u[2:, 1:-1] - u[:-2, 1:-1])**2 / dx -
        (v[1:-1, 2:] - v[1:-1, :-2])**2 / dx)
                     )

    return b
```

Now, to calculate the pressure field, we pass in the original pressure field, the value `b` (which is the result of the `velocity_term` function above) and a target value for difference between two iterates.  We repeatedly update the pressure field until the difference of the L2 norm between two successive iterations is less than that target value.

```python
def pressure_poisson(p, b, l2_target):
    iter_diff = l2_target + 1
    n = 0
    while iter_diff > l2_target and n <= 500:

        pn = p.copy()
        p[1:-1,1:-1] = (.25 * (pn[2:, 1:-1] +
                               pn[:-2, 1:-1] +
                               pn[1:-1, 2:] +
                               pn[1:-1, :-2]) -
                               b[1:-1, 1:-1])

        p[:, 0] = p[:, 1]   #dp/dy = 0 at y = 0
        p[:, -1] = 0        #p = 0 at y = 2
        p[0, :] = p[1, :]   #dp/dx = 0 at x = 0
        p[-1, :] = p[-2, :] #dp/dy = 0 at x = 2

        if n % 10 == 0:
            iter_diff = numpy.sqrt(numpy.sum((p - pn)**2)/numpy.sum(pn**2))

        n += 1

    return p
```

In the interests of brevity, we're only going to worry about the pressure Poisson solver.  The rest of the 2D Navier-Stokes solution is encapsulated in the function `cavity_flow`, which we've prepared ahead of time and saved in a helper file. We just need to import the function:

```python
from snippets.ns_helper import cavity_flow
```

We'll also load up [pickled](https://docs.python.org/2/library/pickle.html) initial conditions, so we can reliably compare final solutions.

```python
import pickle
```

```python
def run_cavity():
    nx = 41
    ny = 41
    with open('IC.pickle', 'rb') as f:
        u, v, p, b = pickle.load(f)

    dx = 2 / (nx - 1)
    dt = .005
    nt = 1000

    u, v, p = cavity_flow(u, v, p, nt, dt, dx,
                         velocity_term,
                         pressure_poisson,
                         rtol=1e-4)

    return u, v, p
```

So what does this all do?  Let's check it out.

```python
u, v, p = run_cavity()
```

```python
%matplotlib inline
from snippets.ns_helper import quiver_plot
```

```python
quiver_plot(u, v, p)
```

#### Save NumPy answers for comparison

```python
with open('numpy_ans.pickle', 'wb') as f:
    pickle.dump((u, v, p), f)
```

Let's profile the `cavity_flow` function and see if there's a specific place that's really hurting our performance.

```python
%timeit run_cavity()
```

```python
%load_ext line_profiler
```

```python
%lprun -f cavity_flow run_cavity()
```

## Where is the bottleneck?

Clearly the PPE is the problem here, so let's use `numba` to rewrite it.

## [Exercise: Speed up the PPE](./exercises/05.Cavity.Flow.Exercises.ipynb#Exercise-1)

```python
from numba import jit
```

```python
# %load snippets/ppe_numba.py
```

```python
u_numba, v_numba, p_numba = run_cavity()
```

```python
assert numpy.allclose(p, p_numba)
assert numpy.allclose(u, u_numba)
assert numpy.allclose(v, v_numba)
```

```python
quiver_plot(u_numba, v_numba, p_numba)
```

```python
%timeit run_cavity()
```

```python
%lprun -f cavity_flow run_cavity()
```

## One more bit of optimization?

```python
@jit(nopython=True)
def velocity_term(b, rho, dt, u, v, dx):
    I, J = b.shape

    for i in range(1, I):
        for j in range(1, J):
            b[i, j] = (
            rho * dx / 16 *
            (2 / dt * (u[i + 1, j] -
                      u[i - 1, j] +
                      v[i, j + 1] -
                      v[i, j - 1]) -
            2 / dx * (u[i, j + 1] - u[i, j - 1]) *
                     (v[i + 1, j] - v[i - 1, j]) -
            (u[i + 1, j] - u[i - 1, j])**2 / dx -
            (v[i, j + 1] - v[i, j - 1])**2 / dx)
            )
    return b
```

## Or be lazy...?

```python
from numba import njit
```

```python
@njit
def pressure_poisson(p, b, l2_target):
    iter_diff = l2_target + 1
    n = 0
    while iter_diff > l2_target and n <= 500:

        pn = p.copy()
        p[1:-1,1:-1] = (.25 * (pn[2:, 1:-1] +
                               pn[:-2, 1:-1] +
                               pn[1:-1, 2:] +
                               pn[1:-1, :-2]) -
                               b[1:-1, 1:-1])

        p[:, 0] = p[:, 1]   #dp/dy = 0 at y = 0
        p[:, -1] = 0        #p = 0 at y = 2
        p[0, :] = p[1, :]   #dp/dx = 0 at x = 0
        p[-1, :] = p[-2, :] #dp/dy = 0 at x = 2

        if n % 10 == 0:
            iter_diff = numpy.sqrt(numpy.sum((p - pn)**2)/numpy.sum(pn**2))

        n += 1

    return p
```

```python
u_numba_lazy, v_numba_lazy, p_numba_lazy = run_cavity()
```

```python
assert numpy.allclose(p, p_numba_lazy)
assert numpy.allclose(u, u_numba_lazy)
assert numpy.allclose(v, v_numba_lazy)
```

```python
%timeit run_cavity()
```