# The Simple Regression Model

## Definition of the Simple Regression Model

$Def\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\EE}[2][\,\!]{\mathbb{E}_{#1}\left[#2\right]}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathrm{N} \left( #1 \right)}
\newcommand{\ow}{\text{otherwise}}$

***Simple Linear Regression Model***: Explains variable $y$ in terms of variable $x$ like the following

$$y = \beta_0 + \beta_1 x + u$$

Here, $y$ is called the ***dependent variable***, or ***explained variable***, or ***response variable***, or ***regressand***; $x$ is the ***independent variable***, or ***explanatory variable***, or ***regressor***. Also, $\beta_0$ is the ***Intercept parameter***, $\beta_1$ is the ***slope parameter***, $u$ is the ***error term***, or ***disturbance***, or ***unobservables***.

Since we already have a intercept, so we can also assume that $\boxed{\EE{u} = 0}$.

**Interpretation**

$\ffrac{\partial y} {\partial x} = \beta_1$ as long as $\ffrac{\partial u} {\partial x} = 0$

- The first term is to say by how much does the dependent variable change if the independent variable is increased by one unit
- remember let all other things remain equal
- rarely applicable though.

***

Then, how $u$ and $x$ are related? The natural way is to consider their *correlation coefficient*, while that's not enough cause it only measures the linear dependence between $u$ and $x$.

$Remark$

>It's possible for $u$ to be uncorrelatd with $x$ while being correlated with a function of $x$, say $x^2$.

A better choice is consider the expected value of $u$ given $x$, which is that *the average value of $u$ does not depend on the value of* $x$, or $u$* is **mean independent** of $x$*. Mathematically:

$$\boxed{\EE{u\mid x} = \EE{u}}$$

Thus we obtain the ***zero conditional mean assumption***: $\EE{u \mid x} = 0$ and then the ***population regression function (PRF)***: $\boxed{ \EE{y \mid x} = \beta_0 + \beta_1 x }$ which is a linear function of $X$.

We call $\beta_0 + \beta_1 x$ the ***systematic part*** of $y$ and $u$, ***unsystematic***.

## Deriving the Ordinary Least Squares Estimates

1. Random sample of $n$ observations: $\CB{\P{x_i,y_i}: i=1,2,\dots, n}$
    - obviously $y_i = \beta_0 + \beta_1 x_i + u_i$ for all $i$
    - $u_i$ is the ***error (disturbance) term*** for observation $i$
2. Fit as good as possible a regression line through the data points

Before that, one significant waypoint: $\EE{u \mid x} = 0 \Longrightarrow \boxed{\Cov{x,u} = 0}\Longrightarrow \boxed{\EE{xu} = 0}$

$Proof$
>$\begin{align}
\EE{u\mid x} &= \int_{-\infty}^{\infty} u\cdot f_{u \mid x} \P{u\mid x} \;\dd{u} \\
&= \int_{-\infty}^{\infty} u\ffrac{f\P{u,x}} {f_{X}\P{x}} \;\dd{u} \\
&= \ffrac{\d{\int_{-\infty}^{\infty} u \cdot f\P{u,x}\;\dd{u}}} {f_X\P{x}} = 0
\end{align}$
>
>$\begin{align}
\Cov{x,u} &= \EE{XU} \\
&= \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} xu\cdot f\P{x,u}\;\dd{x}\dd{u}\\
&= \int_{-\infty}^{\infty}x \cdot{ \int_{-\infty}^{\infty} u\cdot f\P{x,u}\;\dd{u} }\dd{x}
\end{align}$

Alternatively we can write them as $\boxed{\EE{y - \beta_0 - \beta_1 x} = 0}$ and $\boxed{\EE{x\P{y - \beta_0 - \beta_1 x}} = 0}$. And these two *MIGHT* be the restriction on two unknown parameters thus to obtain $\hat\beta_0$ and $\hat\beta_1$, the estimators of $\beta_0$ and $\beta_1$ respectively! The *possible* method is (we can regard the following as moment estimation 矩估计):

$$\begin{cases}
n^{-1} \sum \P{y_i - \hat\beta_0 - \hat\beta_1 x_i} = 0 \\
n^{-1} \sum x_i\P{y_i - \hat\beta_0 - \hat\beta_1 x_i} = 0
\end{cases}$$

the solution is quite easy to obtain 😉😉😉:

$$\hat{\beta}_1 = \ffrac{\d{\sum_{i=1}^{n} \P{x_i - \bar{x}}\P{y_i - \bar{y}}}} {\d{\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}}, \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$$

$Remark$

- $\bar y = n^{-1}\sum_{n} y_i$, the sample average of the $y_i$.
- $\hat\beta_1$ can be interpreted as the sample covariance between $x$ and $y$ divided by the sample variance of $x$.
- Implication: if $x$ and $y$ are positively correlated in the sample, then $\hat\beta_1$ is positive

***
This method is called the ***Ordinary Least Square***. Kinda different from what we've learned before...

$$\begin{cases}
\ffrac{\partial \P{\sum\limits_{i=1}^{n} \P{y_i - \beta_0 - \beta_1 x_i}^2}} {\partial \beta_0} = 0 \Longrightarrow \sum\limits_{i=1}^{n} -2\P{y_i - \hat\beta_0 - \hat\beta_1 x_i} = 0  \\
\ffrac{\partial \P{\sum\limits_{i=1}^{n} \P{y_i - \beta_0 - \beta_1 x_i}^2}} {\partial \beta_1} = 0 \Longrightarrow \sum\limits_{i=1}^{n} -2x_i\P{y_i - \hat\beta_0 - \hat\beta_1 x_i} = 0
\end{cases}$$

- ***fitted value***: $\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$
- ***residual***: $\hat{u}_i = y_i - \hat{y}_i = y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i$

$Remark$

>The result can be obtained by differenting on $\beta_1$ and $\beta_0$. And the regression line will go through point $\P{\bar{x},\bar{y}}$ and different sample will result in different estimates and thus a different regression line. Also, the **residual** $\hat u_i$ is quite different from the **error term** $u_i$.

- ***sum of squared residuals***: $\text{SSR} = \sum\limits_{i=1}^n \hat u_i^2 = \sum\limits_{i=1}^n\P{y_i - \hat\beta_0 - \hat\beta_1 x_i}^2$. And actually the method we've learned before is exactly the way to find the value of $\P{\hat\beta_0, \hat\beta_1}$ to minimize the $\text{SSR}$.
- We also called the first equation set without $n^{-1}$ the ***first order conditions*** for the OLS estimate.
- ***OLS Regression Line***: $\hat y = \hat\beta_0 + \hat\beta_1 x$, also known as ***sample regression function (SRF)*** cause it's the estimated version of *PRF*, $\EE{y \mid x} = \beta_0 + \beta_1$

## Properties of OLS on Any Sample of Data

### algebraic properties of olS Statistics

For any sample data, we have the following

- $\sum\limits_{i=1}^{n} \hat{u}_i = 0$, followed directly from the **first order conditions**
- $\sum\limits_{i=1}^{n} x_i\hat{u}_i = 0$
- $\bar{y} = \hat{\beta}_0 + \hat{\beta}_1 \bar{x}$

from the first one, we can take the average of both sides of equation $y_i = \hat y_i + \hat u_i$ and obtain: $\bar{y} = \bar{\hat y}$; and follow the first and the second one, the sample covariance between $\hat y_i$ and $\hat u_i$ is $0$.

Also we define several measures:

- ***Total sum of squares***: $\text{SST} \equiv \sum\limits_{i=1}^{n} \P{y_i - \bar{y}}^2$
- ***Explained sum of squares***: $\text{SSE} \equiv \sum\limits_{i=1}^{n} \P{\hat{y}_i - \bar{y}}^2$
- ***Residual sum of squares***: $\text{SSR} \equiv \sum\limits_{i=1}^{n} \hat{u}_i^2 = \text{SST} - \text{SSE}$

$Proof$

$\bspace \begin{align}
\text{SST} &= \sum_{i=1}^{n} \P{y_i - \bar y}^2 \\
&= \sum_{i=1}^{n} \SB{\P{y_i - \hat y_i}+ \P{\hat y_i - \bar y}}^2 \\
&= \sum_{i=1}^{n} \SB{\hat u_i + \hat y_i - \bar y}^2 \\
&= \sum_{i=1}^{n} \hat u_i^2 + 2\sum_{i=1}^{n}\hat u_i \P{\hat y_i -\bar y}+\sum_{i=1}^{n}\P{\hat y_i -\bar y}^2\\
&= \text{SSR} + 2\sum_{i=1}^{n}\hat u_i \P{\hat y_i -\bar y} + \text{SSE}
\end{align}$

There $2\sum_{i=1}^{n}\hat u_i \P{\hat y_i -\bar y} = 0$ cause $\ffrac{\sum_{i=1}^{n}\hat u_i \P{\hat y_i -\bar y}} {n-1} = \ffrac{\sum \hat u_i \hat y_i - \sum \hat u_i \bar y} {n-1} = \Cov{\hat u_i, \hat y_i} = \hat\beta_1\Cov{\hat u_i,x_i} = 0$. Thus, proved. Or use other methods if you like.

### Goodness-of-Fit

We first assume that $\text{SST} \neq 0$ which will happen only when all the $y_i$ equal the same value. Then:

$Def$

***Goodness of fit measure*** (***R-squared***), $R^2 = \ffrac{\text{SSE}} {\text{SST}} = 1 - \ffrac{\text{SSR}}{\text{SST}}$.

$Remark$

> R-squared measures the *fraction* of the total variation that is explained by the regression.
>
> And we can also prove that $R^2$ is equal to the square of the sample correlation coefficient between $y_i$ and $\hat y_i$, $\rho^2\P{y_i, \hat y_i}$.
>
> A seemingly low $R^2$ does not necessarily mean that an OLS regression equation is useless. And a high $R^2$ does NOT necessarily mean that the regression has a causal interpretation! (There may be other factors that affect the election outcome)

## Units of Measurement and Functional Form
### The Effects of Changing Units of Measurement on OLS Statistics

The slope parameter and the intercept parameter expand or shrink with the same pace as the measure change of dependent variable, rather, generally only the slope parameter change with the measure change of independent variable.

As for $R^2$, never gonna change with any of $x$ or $y$.

### Incorporating Nonlinearities in Simple Regression

***Semi-logarithmic form***

$\log\P{y} = \beta_0 + \beta_1 x + u$

and here $\beta_1$ is the change rate of "the percentage change of $y$ over $x$"

***Log-logarithmic form***

$\log\P{y} = \beta_0 + \beta_1 \log\P{x} + u$

and here $\beta_1$ is the change rate of "*the percentage change of $y$ over the percentage change of $x$*". The log-log form postulates a constant elasticity model,    whereas the semi-log form assumes a semi-elasticity model

### The Meaning of "linear" Regression

The model (equation: $y = \beta_0 + \beta_1 x + u$) is linear in the *parameters* $\beta_0$ and $\beta_1$.

## Expected Values and Variances of the OLS Estimators

Mainly the statistical properties of OLS estimation. We now view $\hat\beta_0$ and $\hat\beta_1$ as estimators for the parameters $\beta_0$ and $\beta_1$. In other words, the distributions of $\hat\beta_0$ and $\hat\beta_1$ over different random samples from the population.

### Unbiasedness of OLS

$Assumption$ $\text{SLR}.1$ to $\text{SLR}.4$

-  Linearity of the parameters: we define the population model, $y = \beta_0 + \beta_1 x + u$
-  Random sampling: $\CB{\P{x_i,y_i}: i = 1,2,\dots,n}$; and define: $u_i = y_i - \beta_0 - \beta_1 x_i$
-  Sample variation in explanatory variable: $\sum\limits_{i=1}^{n} \P{x_i - \bar{x}}^2 > 0$, or equivalently, sample variance exists, $\CB{x_i, i = 1,2,\dots,n}$ are not all the same value
    - otherwise it would be impossible to study how different values of the explanatory variable lead to different values of the dependent variable
- Zero conditional mean: $\EE{u_i \mid x_i} = 0$

$Theroem.1$ Unbisedness of OLS

Using assumption $\text{SLR}.1$ through $\text{SLR}.4$, we have $\EE{\hat{\beta}_0} = \beta_0 $ and $\EE{\hat{\beta}_1} = \beta_1$.

To prove this, we first mention a missed property for the estimators, linearity.

$$\hat{\beta}_1 = \ffrac{\d{\sum_{i=1}^{n} \P{x_i - \bar{x}}\P{y_i - \bar{y}}}} {\d{\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}} = \ffrac{\d{\sum_{i=1}^{n} \P{x_i - \bar{x}}y_i}} {\d{\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}} \hat= \sum k_i y_i$$

$$k_i = \ffrac{x_i - \bar{x}} {\d{\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}} \Rightarrow\sum k_i = 0,\bspace\sum k_i x_i = \ffrac{\d{\sum_{i=1}^{n}x_i^2 - x_i\bar{x}}} {\d{\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}} = 1$$

$$\hat\beta_1 = \sum k_i y_i = \beta_0 \sum k_i + \beta_1 \sum k_i x_i + \sum k_i u_i = \beta_1 + \sum k_i u_i$$

$$\EE{\hat\beta_1} = \EE{\beta_1 + \sum k_i u_i} = \beta_1 + \sum k_i \EE{u_i} = \beta_1$$

As for $\beta_0$, it's rather easy now.

$$\hat\beta_0 = \bar y - \hat\beta_1 \bar x = \beta_0 + \beta_1 \bar x + \bar{u} -\hat\beta_1 \bar x \Longrightarrow \hat\beta_0 = \beta_0 + \P{\beta_1 - \hat\beta_1} \bar x + \bar{u}$$

And here $\bar u = n^{-1}\sum u_i$.

Also we'd like to mention: $\hat u_i = y_i - \hat\beta_0 -\hat\beta_1 x_i = \P{\beta_0 + \beta_1 x_i + u_i} - \hat\beta_0 -\hat\beta_1 x_i$ so that

$$\hat u_i = u_i - \P{\hat\beta_0 - \beta_0} - \P{\hat\beta_1 - \beta_1}x_i$$

### Variances of the OLS Estimators

$Assumption.5$

Homoskedasticity: $\Var{u_i \mid x_i} = \sigma^2$, called the ***error variance***. This is to say that the value of the explanatory variable must contain no information about the variability of the unobserved factors; and with this we can measure the sample variability, $\Var{\hat\beta_0}$ and $\Var{\hat\beta_1}$

$Remark$

$\text{SLR}.5$ would make the calculation of variance so much easier.

Under that, we first can know that $\EE{u^2 \mid x} = \sigma^2$ and then $\sigma^2 = \EE{u^2 \mid x} \using{\text{assumption } 4}{} \EE{u^2} = \Var{u}$. Then

$Theorem.2$ Sampling Variances of the OLS Estimator

Using $\text{SLR}.1$ to $\text{SLR}.5$,

$\begin{align}
\Var{\hat\beta_1} &= \Var{\beta_1 + \ffrac{\d{\sum_{i=1}^{n} \P{x_i - \bar{x}}u_i}} {\d{\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}}} = \Var{\ffrac{\d{\sum_{i=1}^{n} \P{x_i - \bar{x}}u_i}} {\d{\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}}} \\
&= \P{\ffrac{1} {\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}}^2 \P{\sum_{i=1}^{n} \P{x_i - \bar{x}}^2 \Var{u_i}}\\
&= \ffrac{1} {\SB{{\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}}^2} \sigma^2 \cdot \text{SST}_x = \ffrac{\sigma^2} {\text{SST}_x}
\end{align}$

or even quicker:

$\Var{\hat\beta_1} = \sigma^2 \sum k_i^2 = \ffrac{ \sigma^2} {\d{\sum_{i=1}^{n}\P{x_i - \bar{x}}^2}}$

And since we've already had $\hat\beta_0 = \beta_0 + \P{\beta_1 - \hat\beta_1} \bar x + \bar{u}$, then we have

$\Var{\hat\beta_0} = 0 + \ffrac{\sum\Var{u}} {n^2} + \P{\bar{x}}^2 \cdot \ffrac{\sigma^2} {\text{SST}_x} = \ffrac{ \text{SST}_x} {n\cdot \text{SST}_x}\sigma^2 + \ffrac{ n\P{\bar{x}}^2} {n\cdot \text{SST}_x}\sigma^2 = \ffrac{\sum x_i^2} {n\cdot \text{SST}_x}\sigma^2$

$Remark$

Assuption $4$ and $5$ can be also writen as $\EE{y \mid x} = \beta_0 + \beta_1 x$ and $\Var{y \mid x} = \sigma^2 = \Var{u \mid x} = \Var{u}$ respectively.

In other words, the conditional expectation of $y$ given $x$ is linear in $x$, but the variance of $y$ given $x$ is constant.

$Remark$

When $\Var{u \mid x}$ depends on $x$, the error term is said to exhibit ***heteroskedasticity*** (or just because of nonconstant variance). And since we always have $\Var{u \mid x} = \Var{y \mid x}$, heteroskedasticity is present whenever $\Var{y \mid x}$ is a function of $x$.

$Remark$

1. Larger **error variance** $\sigma^2$, larger $\Var{\hat\beta_1}$
2. More variability in the independent variable $x$ is preferred
***

Also, in case that confidence intervals needed, their ***standard deviation*** are $\text{sd}\P{\hat\beta_i} = \sqrt{\Var{\hat\beta_i}}$

### Estimating the Error Variance

$\bspace \Var{u_i \mid x_i} \using{independency} \sigma^2 = \Var{u_i}$

But Error Variance $\sigma^2$ is not always given thus we have to estimate that sometimes.

An intuitive estimation would be like $\tilde\sigma^2 = \ffrac{1} {n} \sum\limits_{i=1}^{n} \P{\hat{u}_i - \bar{\hat{u}}_i}^2 = \ffrac{1} {n} \sum\limits_{i=1}^{n} \hat{u}_i^2$ however, this is an biased estimation. Here's the unbiased one:

$$\hat\sigma^2 = \ffrac{1} {n-2} \sum_{i=1}^{n} \hat u_i^2$$

$Remark$

>Here $2$ is the number of estimated regression coefficients, or you can call it a degree of freedom...

$Theorem.3$

$\bspace \EE{\hat{\sigma}^2} = \sigma^2$

$Proof$

>We need to use the assumption $\text{SSR}.1$ to $\text{SSR}.5$, and the preceding results: $\sum \hat u_i = 0$ and $\hat u_i = u_i - \P{\hat\beta_0 - \beta_0} - \P{\hat\beta_1 - \beta_1} x_i$. We go across all $i$ for the second equation and will get that: $0 = \bar{u} - \P{\hat\beta_0 - \beta_0} - \P{\hat\beta_1 - \beta_1} \bar x$; substracting it from that again leads to:  $\hat u_i = \P{u_i - \bar u} - \P{\hat\beta_1 - \beta_1} \P{x_i - \bar x}$. Therefore,
>
>$$\bspace \begin{align}
\sum_{i=1}^{n} \hat u_i^2 &= \sum_{i=1}^{n} \P{\P{u_i - \bar u}^2 + \P{\hat\beta_1 - \beta_1}^2\P{x_i - \bar x}^2 - 2\P{u_i - \bar{u}}\P{\hat\beta_1 - \beta_1}\P{x_i - \bar x}} \\
&= \P{\sum_{i=1}^{n} \P{u_i - \bar u}^2} + \P{\hat\beta_1 - \beta_1}^2 \sum_{i=1}^{n} \P{x_i - \bar x}^2 - 2\P{\hat\beta_1 - \beta_1} \sum_{i=1}^{n} u_i\P{x_i - \bar{x}}
\end{align}$$
>
>Taking its expectation, the first term got a expectation $\P{n-1}\sigma^2$; the second term got a expectation of $\EE{\P{\hat\beta_1 - \beta_1}^2} = \Var{\hat\beta_1} = \sigma^2/SST_x$. And for the third term, we first rewrite it as $2\P{\hat\beta_1 - \beta_1}\QQQ$; taking expectation gives $2\sigma^2$. Basically that's it.

Then it's the ***standard error of regression (SER)***: $\hat\sigma = \sqrt{\hat\sigma^2}$. We can also plug the value back to get the estimate value of $\Var{\hat\beta_i}$, standard deviation and standard error of $\hat\beta_i$

$\DeclareMathOperator*{\sd}{sd} \sd\P{{\hat\beta_1}} = \sqrt{{\Var{\hat\beta_1}}} = \sigma / \sqrt{\text{SST}_x}$

$\sd\P{\hat\beta_0} = \sqrt{{\Var{\hat\beta_0}}} = \sqrt{\ffrac{\sum x_i^2} {n\cdot \text{SST}_x}} \sigma$

$\DeclareMathOperator*{\se}{se} \se\P{{\hat\beta_1}} = \sqrt{\widehat{\Var{\hat\beta_1}}} = \hat\sigma / \sqrt{\text{SST}_x}$

$\se\P{\hat\beta_0} = \sqrt{\widehat{\Var{\hat\beta_0}}} = \sqrt{\ffrac{\sum x_i^2} {n\cdot \text{SST}_x}}\hat \sigma$

***


## 6. Nonlinear ensemble filtering for the Lorenz-63 problem

In this notebook, we are interested in the sequential inference

References:

[1] Evensen, G., 1994. Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics. Journal of Geophysical Research: Oceans, 99(C5), pp.10143-10162.

[2] Asch, M., Bocquet, M. and Nodet, M., 2016. Data assimilation: methods, algorithms, and applications. Society for Industrial and Applied Mathematics.

[3] Bishop, C.H., Etherton, B.J. and Majumdar, S.J., 2001. Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Monthly weather review, 129(3), pp.420-436.

[4] Lorenz, E.N., 1963. Deterministic nonperiodic flow. Journal of atmospheric sciences, 20(2), pp.130-141.

[5] Spantini, A., Baptista, R. and Marzouk, Y., 2019. Coupling techniques for nonlinear ensemble filtering. arXiv preprint arXiv:1907.00389.

### The basic steps
To carry out sequential inference in `AdaptiveTransportMap`, we need to carry out a few basic steps:
* **Specify the problem**: Define the state-space model: initial condition, dynamical and observation models (including process and observation noise)
* **Specify the inflation parameters**: Determine the levels of covariance inflation to properly balance the dynamical system and the observations from the truth system
* **Specify the filter**: Choose the ensemble filter to assimilate the observations in the state estimate
* **Perform the sequential inference**: Perform the sequential inference

We will go through all of these here.

```julia
using Revise
using LinearAlgebra
using AdaptiveTransportMap
using Statistics
using Distributions
using OrdinaryDiffEq
```

    ┌ Info: Precompiling AdaptiveTransportMap [bdf749b0-1400-4207-80d3-e689c0e3f03d]
    └ @ Base loading.jl:1278
    ┌ Warning: Type annotations on keyword arguments not currently supported in recipes. Type information has been discarded
    └ @ RecipesBase ~/.julia/packages/RecipesBase/92zOw/src/RecipesBase.jl:116
    ┌ Warning: Type annotations on keyword arguments not currently supported in recipes. Type information has been discarded
    └ @ RecipesBase ~/.julia/packages/RecipesBase/92zOw/src/RecipesBase.jl:116

```julia
using DelimitedFiles
```

Load some packages to make nice figures

```julia
using Plots
default(tickfont = font("CMU Serif", 9),
        titlefont = font("CMU Serif", 14),
        guidefont = font("CMU Serif", 12),
        legendfont = font("CMU Serif", 10),
        grid = false)
# pyplot()

using LaTeXStrings
# PyPlot.rc("text", usetex = "true")
# PyPlot.rc("font", family = "CMU Serif")
# gr()
using ColorSchemes
```

The Lorenz-63  model is a three dimensional system that models the atmospheric convection [4]. This system is a classical benchmark problem in data assimilation. The state $\boldsymbol{x} = (x_1, x_2, x_3)$ is governed by the following set of ordinary differential equations:

\begin{equation}
\begin{aligned}
&\frac{\mathrm{d} x_1}{\mathrm{d} t}=\sigma(x_2-x_1)\\
&\frac{\mathrm{d} x_2}{\mathrm{d} t}=x_1(\rho-x_2)-x_2\\
&\frac{\mathrm{d} x_3}{\mathrm{d} t}=x_1 x_2-\beta x_3,
\end{aligned}
\end{equation}

where $\sigma = 10, \beta = 8/3, \rho = 28$. For these values, the system is chaotic and behaves like a strange attractor. We integrate this system of ODEs with time step $\Delta t_{dyn} = 0.05$. The state is fully observed $h(t,\boldsymbol{x}) = \boldsymbol{x}$ with $\Delta t_{obs}=0.1$. The initial distribution $\pi_{\mathsf{X}_0}$ is the standard Gaussian. The process noise is Gaussian with zero mean and covariance $10^{-4}\boldsymbol{I}_3$. The measurement noise has a Gaussian distribution with zero mean and covariance $\theta^2\boldsymbol{I}_3$ where $\theta^2 = 4.0$.

### Simple twin-experiment

Define the dimension of the state and observation vectors

```julia
Nx = 3
Ny = 3
```

    3

Define the time steps $\Delta t_{dyn}, \Delta t_{obs}$  of the dynamical and observation models. Observations from the truth are assimilated every $\Delta t_{obs}$.

```julia
Δtdyn = 0.05
Δtobs = 0.2
```

    0.2

Define the time span of interest

```julia
t0 = 0.0
tf = 100.0
Tf = ceil(Int64, (tf-t0)/Δtobs)
```

    500

 Define the distribution for the initial condition $\pi_{\mathsf{X}_0}$

```julia
π0 = MvNormal(zeros(Nx), Matrix(1.0*I, Nx, Nx))
```

    FullNormal(
    dim: 3
    μ: [0.0, 0.0, 0.0]
    Σ: [1.0 0.0 0.0; 0.0 1.0 0.0; 0.0 0.0 1.0]
    )

We construct the state-space representation `F` of the system composed of the deterministic part of the dynamical and observation models.

The dynamical model is provided by the right hand side of the ODE to solve. For a system of ODEs, we will prefer an in-place syntax `f(du, u, p, t)`, where `p` are parameters of the model.
We rely on `OrdinaryDiffEq` to integrate the dynamical system with the Tsitouras 5/4 Runge-Kutta method adaptive time marching.

We assume that the state is fully observable, i.e. $h(x, t) = x$.

```julia
h(x, t) = x
F = StateSpace(lorenz63!, h)
```

    StateSpace(AdaptiveTransportMap.lorenz63!, h)

Define the additive inflation for the dynamical and observation models

```julia
### Process and observation noise
σx = 1e-1
σy = 2.0

ϵx = AdditiveInflation(Nx, zeros(Nx), σx)
ϵy = AdditiveInflation(Ny, zeros(Ny), σy)
```

    AdditiveInflation(3, [0.0, 0.0, 0.0], [4.0 0.0 0.0; 0.0 4.0 0.0; 0.0 0.0 4.0], [2.0 0.0 0.0; 0.0 2.0 0.0; 0.0 0.0 2.0])

```julia
ϵx.Σ
```

    3×3 Diagonal{Float64,Array{Float64,1}}:
     1.0e-20   ⋅        ⋅
      ⋅       1.0e-20   ⋅
      ⋅        ⋅       1.0e-20

```julia
model = Model(Nx, Ny, Δtdyn, Δtobs, ϵx, ϵy, π0, 0, 0, 0, F);
```

To perform the nonlinear ensemble filtering, we first need to estimate the transport map $\boldsymbol{S}^{\boldsymbol{\mathcal{X}}}$.

In this notebook, we are going to assume that the basis of features does not change over time, but solely the coefficients $c_{\boldsymbol{\alpha}}$ of the expansion.

To estimate the map, we generate joint samples $(\boldsymbol{y}^i, \boldsymbol{x}^i), \; i = 1, \ldots, N_e$ where $\{\boldsymbol{x}^i\}$ are i.i.d. samples from pushforward of the standard Gaussian distribution by the flow of the Lorenz-63 system.

```julia
# Time span
tspan = (0.0, 100.0)
prob = ODEProblem(lorenz63!, zeros(Nx), tspan)
```

    ODEProblem with uType Array{Float64,1} and tType Float64. In-place: true
    timespan: (0.0, 100.0)
    u0: 3-element Array{Float64,1}:
     0.0
     0.0
     0.0

Set initial condition of the true system

```julia
x0 = rand(model.π0);
```

```julia
data = generate_lorenz63(model, x0, Tf);
```

Initialize the ensemble matrix `X` $\in \mathbb{R}^{(N_y + N_x) \times N_e}$.

```julia
# Ensemble size
Ne = 300

X0 = zeros(model.Ny + model.Nx, Ne)

# Generate the initial conditions for the state.
viewstate(X0, model.Ny, model.Nx) .= rand(model.π0, Ne)
```

    3×300 view(::Array{Float64,2}, 4:6, :) with eltype Float64:
     1.23081    0.604498   -0.153261  …   0.578052   0.698916   0.30136
     0.202271  -0.0696071   0.411907      1.13923   -1.88204    0.204233
     0.177207  -0.323985    0.360247     -1.52934   -0.400928  -1.5781

Use the stochastic ensemble Kalman filter for the spin-up phase. There is no reason to use the stochastic map filter over the first cycles, as the performance of the inference is determined by the quality of the ensemble, not the quality of the filter.

```julia
enkf = StochEnKF(x->x, ϵy, Δtdyn, Δtobs)
```

    Stochastic EnKF  with filtered = false

```julia
Xenkf = seqassim(F, data, Tf, model.ϵx, enkf, deepcopy(X0), model.Ny, model.Nx, t0);
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:08

```julia
tspin = 50.0
Tspin = ceil(Int64, (tspin-t0)/Δtobs)
```

    250

Time average root-mean-squared error

```julia
rmse_enkf = mean(map(i->norm(data.xt[:,i]-mean(Xenkf[i+1]; dims = 2))/sqrt(Nx), Tspin:Tf))
```

    0.6826776463649661

Initialize the structure of the map

```julia
Xspin = vcat(zeros(Ny, Ne), deepcopy(Xenkf[Tspin+1]))
```

    6×300 Array{Float64,2}:
      0.0       0.0       0.0       0.0      …   0.0       0.0       0.0
      0.0       0.0       0.0       0.0          0.0       0.0       0.0
      0.0       0.0       0.0       0.0          0.0       0.0       0.0
      4.50224   2.8974    2.47031   3.56557      3.08375   3.33651   2.98631
      5.40887   3.04887   2.612     4.06651      3.48064   3.6251    3.43293
     20.5057   20.0112   19.4166   20.1042   …  19.6684   20.2709   19.3825

```julia
# Validation of the setup

Xspin = vcat(zeros(Ny, Ne), deepcopy(Xenkf[Tspin+1]))

ystar = data.yt[:,250]
data.tt[250]

ϵx(Xspin, Ny+1, Ny+Nx)

# Compute measurements
observe(F.h, Xspin, tspin, Ny, Nx)

# Perturbation of the measurements
smf.ϵy(Xspin, 1, Ny)
@show norm(Xspin)

# Optimize the map S
optimize(smf.S, Xspin; start = Ny+1)
```

```julia
# Evaluate the transport map
Sx = evaluate(smf.S, Xspin; start = Ny+1)

Xpost = deepcopy(Xspin)
inverse!(Xpost, deepcopy(Sx), smf.S, ystar; start = Ny+1)

@show norm(Xpost[1:Ny,:] .- ystar)

@show norm(evaluate(smf.S, Xspin; start = Ny+1)-evaluate(smf.S, Xpost; start = Ny+1))
```

```julia
tsmf = 100.0
Tsmf = ceil(Int64, (tsmf-tspin)/Δtobs)
```

    250

```julia
p = 2
# order = [[-1], [1; 1], [-1; 1; 0], [-1; 1; 1; 0]]
order = [[-1], [p; p], [-1; p; 0], [-1; p; p; 0]]

# order = [[-1], [-1; -1], [-1; -1; -1], [p; -1; -1 ;p], [-1; p; -1; p; p], [-1; -1; p; p; p; p]]

# parameters of the radial map
γ = 2.0
λ = 0.0
δ = 1e-8
κ = 10.0
β = 1.0

dist = Float64.(metric_lorenz(3))
idx = vcat(collect(1:Ny)',collect(1:Ny)')

smf = SparseRadialSMF(x->x, F.h, β, ϵy, order, γ, λ, δ, κ,
                      Ny, Nx, Ne,
                      Δtdyn, Δtobs,
                      dist, idx; islocalized = true)
```

    SparseRadialSMF(var"#19#20"(), h, 1.0, AdditiveInflation(3, [0.0, 0.0, 0.0], [4.0 0.0 0.0; 0.0 4.0 0.0; 0.0 0.0 4.0], [2.0 0.0 0.0; 0.0 2.0 0.0; 0.0 0.0 2.0]), Sparse Radial Map of dimension Nx = 4 and order p = [[-1], [2, 2], [-1, 2, 0], [-1, 2, 2, 0]]
            with parameters (γ, λ, δ, κ) = (2.0, 0.0, 1.0e-8, 10.0)
    , 3, 3, 0.05, 0.2, [0.0 1.0 1.0; 1.0 0.0 1.0; 1.0 1.0 0.0], [1 2 3; 1 2 3], [0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0], false, true)

```julia
# Ensemble size
Ne = 1000

X0 = zeros(model.Ny + model.Nx, Ne)

# Generate the initial conditions for the state.
viewstate(X0, model.Ny, model.Nx) .= rand(model.π0, Ne)
```

    3×1000 view(::Array{Float64,2}, 4:6, :) with eltype Float64:
     -1.02861   -0.484879   -0.25477   …  -0.211258  -0.323386  -0.853469
     -0.540356  -0.931507    0.198575     -1.71245    0.562809  -0.802693
      0.72809    0.0244415  -0.650659     -1.02144    0.820004   0.444832

```julia
Xsmf = seqassim(F, data, Tf, model.ϵx, smf, X0, model.Ny, model.Nx, 0.0);
```

```julia
rmse_smf = mean(map(i->norm(data.xt[:,i]-mean(Xsmf[i+1]; dims = 2))/sqrt(Nx), Tspin:Tf))
```

    0.5118256249426444

```julia
(rmse_enkf-rmse_smf)/rmse_enkf
```

    0.2502674905675505

```julia
nb = 1
ne = Tf
Δ = 5
plt = plot(layout = grid(3,1), xlim = (-Inf, Inf), ylim = (-Inf, Inf), xlabel = L"t",
           size = (600, 800))

for i =1:3
    plot!(plt[i,1], data.tt[nb:Δ:ne], data.xt[i,nb:Δ:ne], linewidth =  3, color = :teal,
          ylabel = latexstring("x_"*string(i)),
         legend = (i == 1), label = "True")
    plot!(plt[i,1], data.tt[nb:Δ:ne], mean_hist(Xsmf)[i,1+nb:Δ:1+ne], linewidth = 3, grid = false,
          color = :orangered2, linestyle = :dash, label = "Stochastic Map filter")
    scatter!(plt[i,1], data.tt[nb:Δ:ne], data.yt[i,nb:Δ:ne], linewidth = 3, color = :grey,
          markersize = 5, alpha = 0.5, label  = "Observation")
end

plt
```

```julia

```

notebooks/Test nlfilter radial Lorenz 63.ipynb


# Selección óptima de portafolios II

Entonces, tenemos que:
- La LAC describe las posibles selecciones de riesgo-rendimiento entre un activo libre de riesgo y un activo riesgoso.
- Su pendiente es igual al radio de Sharpe del activo riesgoso.
- La asignación óptima de capital para cualquier inversionista es el punto tangente de la curva de indiferencia del inversionista con la LAC.

Para todo lo anterior, supusimos que ya teníamos el portafolio óptimo (activo riesgoso).

En la clase pasada aprendimos a hallar este portafolio óptimo si el conjunto de activos riesgosos estaba conformado únicamente por dos activos:

$$w_{1,EMV}=\frac{(E[r_1]-r_f)\sigma_2^2-(E[r_2]-r_f)\sigma_{12}}{(E[r_2]-r_f)\sigma_1^2+(E[r_1]-r_f)\sigma_2^2-((E[r_1]-r_f)+(E[r_2]-r_f))\sigma_{12}}.$$

- Sin embargo, la complejidad del problema crece considerablemente con el número de variables, y la solución analítica deja de ser viable cuando mencionamos que un portafolio bien diversificado consta aproximadamente de 50-60 activos.
- En esos casos, este problema se soluciona con rutinas numéricas que hagan la optimización por nosotros, porque son una solución viable y escalable a más variables.

**Objetivos:**
- ¿Cuál es el portafolio óptimo de activos riesgosos cuando tenemos más de dos activos?
 - ¿Cómo construir la frontera de mínima varianza cuando tenemos más de dos activos?

*Referencia:*
- Notas del curso "Portfolio Selection and Risk Management", Rice University, disponible en Coursera.
___

## 1. Maximizando el radio de Sharpe

### ¿Qué pasa si tenemos más de dos activos riesgosos?

En realidad es algo muy similar a lo que teníamos con dos activos.
- Para dos activos, construir la frontera de mínima varianza es trivial: todas las posibles combinaciones.
- Con más de dos activos, recordar la definición: la frontera de mínima varianza es el lugar geométrico de los portafolios que proveen el mínimo riesgo para un nivel de rendimiento dado.

 Ver en el tablero.

Analíticamente:
- $N$ activos,
- caracterizados por $(\sigma_i,E[r_i])$,
- cada uno con peso $w_i$, con $i=1,2,\dots,N$.

Entonces, buscamos los pesos tales que

\begin{align}
\min_{w_1,\dots,w_n} & \quad \sum_{i=1}^{N}w_i^2\sigma_i^2+\sum_{i=1}^{N}\sum_{j=1,j\neq i}^{N}w_iw_j\sigma_{ij}\\
\text{s.a.}          & \quad \sum_{i=1}^{N}w_i=1, w_i\geq0\\
                     & \quad \sum_{i=1}^{N}w_iE[r_i]=\bar{\mu},
\end{align}

donde $\bar{\mu}$ corresponde a un nivel de rendimiento objetivo.

**Obviamente, tendríamos que resolver este problema para muchos niveles de rendimiento objetivo.**

-  Explicar relación con gráfica.

-  Recordar clase 10.

Lo anterior se puede escribir vectorialmente como:

\begin{align}
\min_{\boldsymbol{w}} & \quad \boldsymbol{w}^T\Sigma\boldsymbol{w}\\
\text{s.a.}          & \quad \boldsymbol{1}^T\boldsymbol{w}=1, \boldsymbol{w}\geq0\\
                     & \quad E[\boldsymbol{r}^T]\boldsymbol{w}=\bar{\mu},
\end{align}

donde:
- $\boldsymbol{w}=\left[w_1,\dots,w_N\right]^T$ es el vector de pesos,
- $\boldsymbol{1}=\left[1,\dots,1\right]^T$ es un vector de unos,
- $E[\boldsymbol{r}]=\left[E[r_1],\dots,E[r_N]\right]^T$ es el vector de rendimientos esperados, y
- $\Sigma=\left[\begin{array}{cccc}\sigma_{1}^2 & \sigma_{12}  & \dots  & \sigma_{1N} \\
                                   \sigma_{21}  & \sigma_{2}^2 & \dots  & \sigma_{2N} \\
                                   \vdots       & \vdots       & \ddots & \vdots      \\
                                   \sigma_{N1}  & \sigma_{N2}  & \dots  & \sigma_{N}^2\end{array}\right]$ es la matriz de varianza-covarianza.

**Esta última forma es la que comúnmente usamos al programar, por ser eficiente y escalable a problemas de N variables.**

### Entonces, ¿para cuántos niveles de rendimiento objetivo tendríamos que resolver el anterior problema con el fin de graficar la frontera de mínima varianza?
- Observar que el problema puede volverse muy pesado a medida que incrementamos el número de activos en nuestro portafolio...
- Una tarea bastante compleja.

### Sucede que, en realidad, sólo necesitamos conocer dos portafolios que estén sobre la *frontera de mínima varianza*.
- Si logramos encontrar dos portafolios sobre la frontera, entonces podemos a la vez encontrar todas las posibles combinaciones de estos dos portafolios para trazar la frontera de mínima varianza.
- Ver el caso de dos activos.

### ¿Qué portafolios usar?
Hasta ahora, hemos estudiando profundamente como hallar dos portafolios muy importantes que de hecho yacen sobre la frontera de mínima varianza:
1. Portafolio de EMV: máximo SR.
2. Portafolio de mínima varianza: básicamente, el mismo problema anterior, sin la restricción de rendimiento objetivo.

Luego, tomar todas las posibles combinaciones de dichos portafolios usando las fórmulas para dos activos de medias y varianzas:
- w: peso para el portafolio EMV,
- 1-w: peso para le portafolio de mínima varianza.

## 2. Ejemplo ilustrativo.

Retomamos el ejemplo de mercados de acciones en los países integrantes del $G5$: EU, RU, Francia, Alemania y Japón.

```python
# Importamos pandas y numpy
import pandas as pd
import numpy as np
```

```python
# Resumen en base anual de rendimientos esperados y volatilidades
annual_ret_summ = pd.DataFrame(columns=['EU', 'RU', 'Francia', 'Alemania', 'Japon'], index=['Media', 'Volatilidad'])
annual_ret_summ.loc['Media'] = np.array([0.1355, 0.1589, 0.1519, 0.1435, 0.1497])
annual_ret_summ.loc['Volatilidad'] = np.array([0.1535, 0.2430, 0.2324, 0.2038, 0.2298])

annual_ret_summ.round(4)
```

      EU
      RU
      Francia
      Alemania
      Japon

      Media
      0.1355
      0.1589
      0.1519
      0.1435
      0.1497

      Volatilidad
      0.1535
      0.243
      0.2324
      0.2038
      0.2298

```python
# Matriz de correlación
corr = pd.DataFrame(data= np.array([[1.0000, 0.5003, 0.4398, 0.3681, 0.2663],
                                    [0.5003, 1.0000, 0.5420, 0.4265, 0.3581],
                                    [0.4398, 0.5420, 1.0000, 0.6032, 0.3923],
                                    [0.3681, 0.4265, 0.6032, 1.0000, 0.3663],
                                    [0.2663, 0.3581, 0.3923, 0.3663, 1.0000]]),
                    columns=annual_ret_summ.columns, index=annual_ret_summ.columns)
corr.round(4)
```

      EU
      RU
      Francia
      Alemania
      Japon

      EU
      1.0000
      0.5003
      0.4398
      0.3681
      0.2663

      RU
      0.5003
      1.0000
      0.5420
      0.4265
      0.3581

      Francia
      0.4398
      0.5420
      1.0000
      0.6032
      0.3923

      Alemania
      0.3681
      0.4265
      0.6032
      1.0000
      0.3663

      Japon
      0.2663
      0.3581
      0.3923
      0.3663
      1.0000

```python
# Tasa libre de riesgo
rf = 0.05
```

Esta vez, supondremos que tenemos disponibles todos los mercados de acciones y el activo libre de riesgo.

#### 1. Construir la frontera de mínima varianza

##### 1.1. Encontrar portafolio de mínima varianza

```python
# Importamos funcion minimize del modulo optimize de scipy

```

```python
## Construcción de parámetros
# 1. Sigma: matriz de varianza-covarianza

# 2. Eind: rendimientos esperados activos individuales

```

```python
# Función objetivo

```

```python
# Dato inicial

# Cotas de las variables

# Restricciones

```

```python
# Portafolio de mínima varianza

```

```python
# Pesos, rendimiento y riesgo del portafolio de mínima varianza

```

##### 1.2. Encontrar portafolio EMV

```python
# Función objetivo

```

```python
# Dato inicial

# Cotas de las variables

# Restricciones

```

```python
# Portafolio EMV

```

```python
# Pesos, rendimiento y riesgo del portafolio EMV

```

##### 1.3. Construir frontera de mínima varianza

También debemos encontrar la covarianza (o correlación) entre estos dos portafolios:

```python
# Covarianza entre los portafolios

```

```python
# Correlación entre los portafolios

```

```python
# Vector de w

```

```python
# DataFrame de portafolios:
# 1. Índice: i
# 2. Columnas 1-2: w, 1-w
# 3. Columnas 3-4: E[r], sigma
# 4. Columna 5: Sharpe ratio

#frontera
```

```python
# Importar librerías de gráficos

```

```python
# Gráfica de dispersión de puntos coloreando
# de acuerdo a SR, los activos individuales
# y los portafolios hallados

```

**A partir de lo anterior, solo restaría construir la LAC y elegir la distribución de capital de acuerdo a las preferencias (aversión al riesgo).**
___

## 3. Comentarios finales

### 3.1. Restricciones adicionales

Los inversionistas pueden tener restricciones adicionales:
1. Restricciones en posiciones cortas.
2. Pueden requerir un rendimiento mínimo.
3. Inversión socialmente responsable: prescinden de inversiones en negocios o paises considerados éticamente o políticamente indeseables.

Todo lo anterior se puede incluir como restricciones en el problema de optimización, y puede ser llevado a cabo a costa de un cociente de Sharpe menor.

### 3.2. Críticas a la optimización media varianza
1. Solo importan medias y varianzas: recordar que la varianza subestima el riesgo en algunos casos.
2. Preferencias media-varianza tratan las ganancias y pérdidas simétricamente: el sentimiento de insatisfacción de una perdida es mayor al sentimiento de satisfacción de una ganancia (aversión a pérdidas).
3. La aversión al riesgo es constante: la actitud frente al riesgo puede cambiar, por ejemplo con el estado de la economía.
4. Horizonte corto (un periodo).
5. Basura entra - basura sale: la optimización media varianza es supremamente sensible a las entradas: estimaciones de rendimientos esperados y varianzas.
___

# Anuncios parroquiales

## 1. Revisar archivo Tarea 6.
## 2. Fin de módulo. Abrir Clase0 y revisar objetivos del módulo.
## 3. [Nota interesante](http://yetanothermathprogrammingconsultant.blogspot.com/2016/08/portfolio-optimization-maximize-sharpe.html)

Created with Jupyter by Esteban Jiménez Rodríguez.