# Forecasting U.S. Dollar Future prices using a VAR model

Alexandru Giurca, VU Amsterdam, November 2019

**Summary**

One of the main needs when we talk about trading is to make some kind of forecast about the future status of the instrument operated so that we can position our offers. As one of the objectives of the analysis of multivariate time series is the realization of forecasts, in this activity I will implement a model of this class called Vector Auto Regressive (VAR). I will fit the model to the DOLLAR book information.

## 1. Introduction

From a trading point of view, we need to make decisions about whether to hold a position or not, buy, sell or even do nothing. If we have a time series of observations related to the variable of interest (e.g. instrument or spread trade) and if these data contain information about its dynamics, it is reasonable to use some function of the data already collected, for the forecast, of way that:

$$\hat{y}_{K,\, T+1} = f \left(y_{1, T}, y_{2, T}, y_{1, T-1}, \dots \right)$$

where $ y_{k, t} $ denotes a random variable, $ t $ denotes its index over time and $ k $, the index of the variable itself. The forecast equation above implies that there may be interdependence between the variables used. These multiple variables can be the returns of different assets, macroeconomic variables etc.

Since the time series is a set of random variables, we assume that this series is generated by a stochastic process with an associated  probability space $ \left(\Omega, \, \mathscr{F}, \, P \right) $, where $\Omega $ is the sample space, $ \mathscr{F}$ or $ \sigma$-algebra represents all possible subsets of $ \Omega $ and $ P $, the probability measure. We define that a random variable is a function that associates elements of a $ \Omega $ sample space with values in the set of real numbers.

Thus, assuming that our data are realizations of random variables, that the same data-generating process prevails over the entire $ T $ period (although not known) and that linear functions (in the parameters) are relatively easy to deal with, we can write a vector auto-regressive process so that:
$$y_t = \mu + A_q y_{t-1} +  \dots + A_p y_{t-p} + u_t$$

where $ y_t: = \left(y_ {1t}, \dots, y_ {Kt} \right)'$ is a vector of random variables, $ \mu:= \left(\mu_{t}, \dots, \mu_{K} \right)'$ is a vector of constants, $ A $ is a square matrix of parameters and $u_t = \left(u_{1t}, \dots, u_{Kt} \right)$ is a string dops forecast errors, taken as iid with expectation $ 0 $. This is equivalent to saying that all the useful information contained in $ \sigma$-algebra has been incorporated into $ y_t $ and there are no systematic errors in the forecasts.

In this work I will estimate the parameters of the model above, which is also called the autoregressive vector model (VAR). Since it is an AR, it is expected that the data used as input will be weakly stationary (first and second invariant moments), respecitvely all eigenvalues of $ A $ have a module less than 1).

When choosing the model, model order selection will be performed. This involves selecting a model order that minimizes one or more information criteria evaluated over a range of model orders. Commonly used information criteria include, Akaike Information Criterion (AIC), Schwarz-Bayes Criterion (SBC) – also known as the Bayesian Information Criterion (BIC) – Akaike’s Final Prediction Error Criterion (FPE), and Hannan-Quinn Criterion (HQ).

## 2. Model Implementation

In this session I will dive into the details of the parameter estimation process and compute the selection criteria.

### 2.1. Parameter Estimation

Since the model was described as a linear relationship between the variables, we can estimate the model parameters by minimizing the sum of square errors. First, assuming that the time series $ y_1, ..., y_T $ of size $T$ and "width" $K$ (the number of variables available - lags), we have:

```python
import pandas as pd
df = pd.read_excel('data/inputs_outputs.xlsx', header=None)
l_index = pd.date_range(start= '01/10/2016', end='01/10/2019')
df.index = l_index[:df.shape[0]]
print(df.head())

ax = df.plot(legend=False)
ax.set_title('Data for Tests \n', fontsize=16)
```

```python
df_original = df.copy()
```

We assume that the data used is stationary (which seems reasonable by looking at the graph). There seems to be no seasonality, trend, variance increase etc.

$$
\begin{aligned}
Y &:= \left(  y_1, \dots, y_T \right)  \;\;\;\;\; &\left(K \times T \right)\\
B &:= \left(  \mu,\, A_1 \dots, A_p \right)  \;\;\;\;\; &\left(K \times (Kp + 1) \right)\\
Z_t &:= \left(  1, y_t, \dots, y_{t-p+1} \right)'  \;\;\;\;\; &\left(1 \times (Kp + 1) \right)\\
Z &:= \left(  Z_0, \dots, Z_T \right)  \;\;\;\;\; &\left((Kp + 1) \times T \right)
\end{aligned}
$$

We rewrite the var as being of the form $ Y = B Z + U $, implying that the residuals are of the form $ U = Y - BZ $. To determine the least squares estimator (see \textsc{Econometric Methods with Application in Business and Economics}), we write the sum of the residual squares as:
$$
\begin{aligned}
S(b) &= \sum U^2 = U'U = \left( Y - BZ \right)' \left( Y - BZ \right)\\
     &= Y'Y - Y'BZ - B'Z'Y + B'Z'ZB\\
\end{aligned}
$$

As $ Y'ZB = B'Z'Y $, we can rewrite the above equation as

$$S(b) = Y'Y - 2B'Z'Y + B'Z'ZB$$

Taking the derivative with respect to $B$ and setting equal to 0, we obtain

$$
\begin{aligned}
S(b) &= \sum U^2 = U'U = \left( Y - BZ \right)' \left( Y - BZ \right)\\
     &= Y'Y - Y'BZ - B'Z'Y + B'Z'ZB\\
\end{aligned}
$$

A file called *var_model* implementing this model was created in the repository. The values will be compared with the values obtained with the implementation from the library *statsmodels* from Python.

```python
import var_model.vector_autoregression as var
self = var.VectorAutoregression(df)
self.fit(3)
```

```python
# print matrices A
import pandas as pd
for idx, A in enumerate(self.na_A):
    print("A{}:\n {}".format(idx + 1, pd.DataFrame(A)))
```

    A1:
               0         1         2         3         4
    0 -0.043374  0.128951 -0.127479  0.049648 -0.071250
    1  0.005321  0.178780  0.003759  0.032509 -0.072283
    2  0.008242  0.115180 -0.050708  0.007212 -0.041946
    3  0.035804  0.166959  0.048654 -0.117873 -0.049864
    4 -0.064091 -0.059743 -0.097786  0.268141 -0.054190
    A2:
               0         1         2         3         4
    0  0.013431  0.001495 -0.242662  0.245055 -0.075217
    1 -0.244118 -0.004717 -0.345717  0.413840 -0.032270
    2 -0.047898  0.179970 -0.487353  0.295174  0.055272
    3 -0.043729  0.159101 -0.365092  0.193884  0.068610
    4 -0.028104  0.096378 -0.391956  0.162951 -0.030571
    A3:
               0         1         2         3         4
    0 -0.044115  0.015434  0.056729 -0.180469  0.065909
    1 -0.075600 -0.076008  0.070572  0.017278  0.042417
    2 -0.110870  0.027031 -0.034428 -0.013438 -0.022812
    3 -0.040539  0.032051 -0.016870 -0.037124 -0.001811
    4 -0.046125 -0.081289  0.247950 -0.131097 -0.025088

The matrix $ \Sigma_U $ was estimated directly by the definition $ \tilde{\Sigma}_U = \mathbf{E} \left[u_t u_t '\right] $, that is, I calculated the matrix $ U = Y - BZ $ and computed the outer product for its transposed and divided it by T. Then I adjusted the result by the model's degree of freedom, so that $ \hat{\Sigma}_{U} =\frac{T}{T - Kp -1} \tilde{\Sigma}_{U}$.

```python
import pandas as pd
print("Sigma_U:\n {}".format(pd.DataFrame(self.na_Sigma)))
```

    Sigma_U:
               0         1         2         3         4
    0  0.000195  0.000068  0.000056  0.000077  0.000064
    1  0.000068  0.000283  0.000154  0.000166  0.000117
    2  0.000056  0.000154  0.000309  0.000255  0.000126
    3  0.000077  0.000166  0.000255  0.000267  0.000130
    4  0.000064  0.000117  0.000126  0.000130  0.000461

Now, compare these values with the values obtained from *statsmodels*.

```python
from statsmodels.tsa.api import VAR
import statsmodels.tsa.vector_ar.util as util
model = VAR(df)
results = model.fit(3)
aux = util.get_var_endog(df.values, 3)
```

```python
# Compute Errors
print("error in Z: {:0.8f}".format(sum(sum(abs(self.na_Z.T - aux)))))
print("error in A: {:0.8f}".format(sum(sum(sum(abs(self.na_A - results.coefs))))))
print("error in B: {:0.8f}".format(sum(sum(abs(self.na_betahat - results.params.values)))))
print("error in Sigma_U: {:0.8f}".format(sum(sum(abs(self.na_Sigma - results.sigma_u.values)))))
```

    error in Z: 0.00000000
    error in A: 0.00000000
    error in B: 0.00000000
    error in Sigma_U: 0.00000000

As can be seen, up to the eighth decimal, the values obtained were identical to those obtained by the benchmark library.

### 2.2. Forecast

To state what will happen in the future $\left(y_1, ..., y_k \right) $, with a process $VAR(p)$ adjusted to a data set $ \Omega_t = \{y_s \mid s\leq t\} $ for a $ h $ time horizon, we need to determine which is the optimal *forecast* by determining the one that minimizes a cost function associated with its errors (mean squares - MSE). The predictor that minimizes these errors is the conditional expectation (Lutkepohl, p. 33)

$$\mathbf{E}\left[ y_{t+h} \right] := \mathbf{E}\left[ y_{t+h} \mid  \Omega_t \right] = \mathbf{E}\left[ y_{t+h} \mid  \{ y_s \mid s \leq t \} \right]$$

$$
\mathbf{E}\left[ y_{t+1} \right ] = \mu + A_1 y_t + ... + A_p y_{t-p + 1} \\
\mathbf{E}\left[ y_{t+2} \right ] = \mu + A_1 \mathbf{E}\left[ y_{t+1} \right] + A_2 y_t + ... + A_p y_{t-p + 2} \\
\vdots
$$

Applying this method and comparing it with the benchmark implementation, we arrive that

```python
my_forecast, my_max, my_min = self.forecast(df[-3:].values, 1)
na_forecast = results.forecast_interval(df[-3:].values, 1)
```

```python
# One step Ahead forecast for each variable
print("error in Forecast: {:0.8f}\n".format((sum(abs(my_forecast - na_forecast[0][0])))))
print(pd.Series(my_forecast))
```

    error in Forecast: 0.00000000

    0    0.001801
    1    0.006724
    2    0.006905
    3    0.005442
    4    0.000813
    dtype: float64

### 2.3. Confidence Intervals

The prediction error (and consequently, the confidence interval) can be obtained through the error variance-covariance matrix (MSE matrix). However, when you want to perform *forecasts* for more than one period, you must also define the Moving Average(MA) coefficient matrix, which for a VAR(2) is

$$
\phi_1 = A_1 \\
\phi_2 = \phi_1 A_1 + A2 \\
\phi_3 = \phi_2 A_1 + \phi_1 A2 \\
\vdots \\
\phi_i = \phi_{i-1} A_1 + \phi_{i-2} A_2
$$

Thus, the *forecast* MSE matrix is obtained recursively by applying

$$
\Gamma_y(0) = \Sigma_y = \sum_{i=0}^{\infty} \phi_i \Sigma_u \phi_i^{'}
$$

$$
\Sigma_y (1) = \Sigma_u \\
\Sigma_y (2) = \Sigma_u + \phi_{1} \Sigma_u \phi_{1}' \\
\Sigma_y (3) = \Sigma_y (2) + \phi_{2} \Sigma_u \phi_{2}' \\
\vdots
$$

The $ h $ in $ \Sigma_y(h) $ refers to how many steps ahead (forward in future time) the forecast function applies. Since the VAR model assumes that $ u_t \sim N \left (0, \, \Sigma_y(h) \right) $ errors, we can assume that the *forecast* error is also normally distributed. With this assumption, we can define a confidence interval of the form

$$ \left[y_{k, \, t}(h) - z_{(\alpha /2)} \sigma_k(h), \, \, \, \, \, y_{k, \, t}(h) + z_{(\alpha / 2)} \sigma_k (h) \right] $$

Where $ \sigma(h) $ is the square root of the ** *k-th* diagonal element ** $ \Sigma_y(h) $:

$$
\Sigma_y(1) =
\begin{pmatrix}
  \textbf{2.25} & 0 & 0 \\
  0 & \textbf{1.0} & .5 \\
  0 & .5 & \textbf{.74}
\end{pmatrix}
$$

Confidence intervals are given by

$$
y_{1, t} (1) \pm z_{(\alpha / 2)} \sqrt{2.25} \\
y_{2, t} (1) \pm z_{(\alpha / 2)} \sqrt{1.0} \\
y_{3, t} (1) \pm z_{(\alpha / 2)} \sqrt{.74}
$$

So, let's calculate the confidence intervals for 3 periods ahead and compare with the implementation of the *benchmark*.

```python
import var_model.vector_autoregression as var
import numpy as np
import pandas as pd
self = var.VectorAutoregression(df)
self.fit(3)
```

```python
my_forecast, my_max, my_min = self.forecast(df[-5:].values, 3)
na_forecast, na_min, na_max = results.forecast_interval(df[-3:].values, 3)
```

```python
print("Lower bound error: {:0.7f}".format(sum(abs(na_min[-1] - my_min))))
print("Forecast Error: {:0.7f}".format(sum(abs(na_forecast[-1] - my_forecast))))
print("Upper bound error: {:0.7f}".format(sum(abs(na_max[-1] - my_max))))
print('\n\n')
df_plot = pd.DataFrame([my_min, na_min[-1]], index=['my model', 'statsmodels'])
df_plot.index.name = 'Minimum'
print(str(df_plot))
print('')

df_plot = pd.DataFrame([my_max, na_max[-1]], index=['my model', 'statsmodel'])
df_plot.index.name = 'Maximum'
print(df_plot)
```

    Lower bound error: 0.0000000
    Forecast Error: 0.0000000
    Upper bound error: 0.0000000

                        0         1         2         3         4
    Minimum
    my model    -0.030019 -0.035569 -0.037285 -0.034497 -0.046392
    statsmodels -0.030019 -0.035569 -0.037285 -0.034497 -0.046392

                       0         1         2         3         4
    Maximum
    my model    0.026364  0.033363  0.034616  0.032315  0.040077
    statsmodel  0.026364  0.033363  0.034616  0.032315  0.040077

Although the *forecast* was the same as the implementation of *statsmodels*, the intervals found were slightly different. However, the error in relation to the implementation of the *benchmark* was symmetrical, which suggests that they used a different estimation method than the one implemented here. The dcumentation reveals that the inverse Hessian

### 2.4. Selecting lag order for VAR

The order has to be chosen so that the projections have the best precision (or the smallest mean square error). There is no single way to do this. One option is to use the final prediction error (FPE) criterion, defined as

$$FPE(m) = \left[ \frac{T + Km + 1}{T - Km -1} \right]^{K} \left |  \Sigma_u(m) \right | $$

Another criterion is The Akaike Information Criterion (AIC), defined as

$$AIC(m) = \ln \left| \Sigma_u(m) \right| + \frac{2mK^2}{T}$$

A third option is the Hannan-Quinn Criterion, which has interesting sample properties (consistency, in this case):

$$HQ(m) = \ln \left|  \Sigma_u(m) \right| + \frac{2\ln \ln T}{T}mK^2$$

The last option implemented will be another one with the same asymptotic property of the previous criterion. Schwarz's criterion (SC or BIC):

$$SC(m) = BIC(m) = \ln \left|  \Sigma_u(m) \right| + \frac{\ln T}{T}mK^2$$

```python
import var_model.vector_autoregression as var
import numpy as np
import pandas as pd
self = var.VectorAutoregression(df)
```

```python
self.select_order(10)
```

    Ordem com menor valor para cada Critério:
      Criterion FPE:  		Ord. 1
      Criterion AIC:  		Ord. 6
      Criterion HQ:  		Ord. 1
      Criterion SC(BIC):  		Ord. 1

           FPE      AIC       HQ  SC(BIC)
    Ordem
    1      0.0 -10.5605 -10.5571 -10.5508
    2      0.0 -10.5590 -10.5522 -10.5395
    3      0.0 -10.5612 -10.5511 -10.5320
    4      0.0 -10.5618 -10.5483 -10.5229
    5      0.0 -10.5642 -10.5473 -10.5155
    6      0.0 -10.5651 -10.5449 -10.5068
    7      0.0 -10.5649 -10.5413 -10.4967
    8      0.0 -10.5632 -10.5362 -10.4853
    9      0.0 -10.5596 -10.5293 -10.4720
    10     0.0 -10.5569 -10.5232 -10.4596

In this case, we would take the order 1. The autocorrelation function reveals that residuals are white noise:

```python
model = VAR(df)
results_1 = model.fit(1)
results_1.plot_acorr()
```

### 2.5 Impulse Response Analysis

Impulse response analysis is an important step in econometric analyses. Their main purpose is to describe the evolution of a model’s variables in reaction to a shock in one or more variables. This feature allows to trace the transmission of a single shock within an otherwise noisy system of equations and, thus, makes them very useful tools in the assessment of economic policies.

Since all variables in a VAR model depend on each other, individual coefficient estimates only provide limited information on the reaction of the system to a shock. In order to get a better picture of the model’s dynamic behaviour, impulse responses (IR) are used. The departure point of every impluse reponse function for a linear VAR model is its moving average (MA) representation, which is also the Forecast Error Impulse Response (FEIR) function. The FEIR $\Phi_i$ for the ith period after the shock is obtained by

$$\Phi_i= \sum_{j=1}^{i} \Phi_{i−j}A_j,  i=1,2,...$$

with $\Phi_0=I_K$ and $A_j=0$ for $j>p$, where $K$ is the number of endogenous variables and $p$ is the lag order of the VAR model.

```python
irf = results.irf(10)
irf.plot(orth=False)
```

A caveat of FEIRs is that they cannot be used to assess contemporaneous reactions of variables. This can be explained by the fact that for the construction of $\Phi_i$ only the coefficient matrices $A_j$ are used, which contain no information on contemporaneous relationships.

Correlation matrices only describe the correlation between the errors, but it remains unclear in which direction the causalities go. Identifying these causal relationships is one of the main challenges of any VAR analysis.

#### 2.5.1 Orthogonal Impulse Responses

Regardless of the specific methods used to identify the shocks of a linear VAR model, further information on contemporaneous relationships can be introduced to the FEIR by simply multiplying it by a matrix $F$:
$$\Theta_i = \Phi_i F \ for \ i=0,1, \dots$$

A common approach to identify the shocks of a VAR model is to use orthogonal impulse respones (OIR). The basic idea is to decompose the variance-covariance matrix so that \Sigma = PP^{\prime}, where $P$ is a lower triangular matrix with positve diagonal elements, which is often obtained by a Choleski decomposition.

The corresponding orthongonal impulse response function is then
$$\Theta^o_i = \Phi_i P$$

#### 2.5.2 Forecast Error Variance Decomposition (FEVD)

Forecast errors of component j on k in an i-step ahead forecast can be decomposed using the orthogonalized impulse responses $\Theta_i$:

\begin{align}
\begin{aligned}
\omega_{jk, i} = \sum_{i=0}^{h-1} (e_j^\prime \Theta_i e_k)^2 / \mathrm{MSE}_j(h)\\\mathrm{MSE}_j(h) = \sum_{i=0}^{h-1} e_j^\prime \Phi_i \Sigma_u \Phi_i^\prime e_j
\end{aligned}
\end{align}

For the five step ahead:

```python
fevd = results.fevd(5)
fevd.summary()
results.fevd(20).plot()
```

## 3. Applying the VAR model

In this section I will apply the model to the high frequency Dolar future data. Data will be grouped in intervals of 10 seconds. Rather than fitting the model to returns, the model is applied to the first difference of the selected variables.

### 3.1. Order Flow Imbalance

Trading book events - market orders, limit and cancellations orders - impact the asset price dynamics. This impact can be modeled by a single variable: the *order flow imbalance* (OFI), which represents the *net* of the quantity that entered and left the bid and ask queues. Thus, for each event $ E_n$:

$$e_n = \mathbb{1}_{P_{n}^{B} \geq P_{n-1}^{B}} q^{B}_{n} - \mathbb{1}_{P_{n}^{B} \leq P_{n-1}^{B}}  q^{B}_{n-1} - \mathbb{1}_{P_{n}^{A} \leq P_{n-1}^{A}} q^{A}_{n} + \mathbb{1}_{P_{n}^{A} \geq P_{n-1}^{A}}  q^{A}_{n-1}$$

Where $ q^{B}_{n} $, $ q^{A}_{n} $ refer to the quantity at the best price in Bid and Ask, respectively in the current period $ n $. The subscript $ n-1 $ refers to the previous state. Since events affecting the book happen at random $ \tau_n $ times, we define $ N(t) = \max\{n \mid \tau_n \leq t \} $ the number of events that occurred between $ \left[0 , \, t \right] $. The OFI variable is defined over the interval $ \left[t_{k-1}, \, t_k \right] $ and is the sum of each $ e_n $ that occurred in this period:

$$OFI_k = \sum^{N(t_k)}_{n=N(t_{k-1})+1} e_n$$

This variable can explain the change in mid-price in short periods of time using a simple linear relationship (*The Price Impact of Order Book Events*). The intuition, according is that "it needs volume to move the price". The change in mid-price, in their study, is defined in ticks:

$$\Delta P_k = \frac{\left( P_k - P_{k-1}\right)}{\delta}$$

where $ \delta $ is the tick size of the analyzed instrument.

### 3.2. Computing the OFI

Let's start by computing the OFI for two days of Level I data (http://www.investopedia.com/terms/l/level1.asp) from the contract book. First, let's check the file size

```python
import var_model.vector_autoregression as var
import numpy as np
import pandas as pd
```

```python
import zipfile
s_fname = "data/dol.zip"
archive = zipfile.ZipFile(s_fname, 'r')
def foo():
    f_total = 0.
    f_tot_rows = 0.
    for i, x in enumerate(archive.infolist()):
        f_total += x.file_size/ 1024.**2
        for num_rows, row in enumerate(archive.open(x)):
            f_tot_rows += 1
        print("{}:\t{:,.0f} rows\t{:0.2f} MB".format(x.filename, num_rows + 1, x.file_size/ 1024.**2))

%time foo()
```

    dol.txt:	756,137 rows	24.60 MB
    Wall time: 1.35 s

Now, let's calculate line by line the value of $ e_n $ and cumulate it for 10 seconds. Each time I pass this time, I will print the cumulated data in a new file and reset the values. I will also pair this data with the change in the asset's mid-price, already standardized by the tick-size (half point). I will also save the return log in these 10 seconds.

```python
def measure_e_n(row, last_best):
    '''
    Measure the e_n of the current event
    :param row: dictionary. current row from the file
    :param last_best: tuple. best price and best quantity
    '''
    e_n = 0
    if row['Type'] == 'BID':
        e_n += (row['Price'] >= last_best[0]) * row['Size']
        e_n -= (row['Price'] <= last_best[0]) * last_best[1]
    elif row['Type'] == 'ASK':
        e_n -= (row['Price'] <= last_best[0]) * row['Size']
        e_n += (row['Price'] >= last_best[0]) * last_best[1]
    return e_n

def convert_float_to_time(f_time):
    '''
    Converst number of seconds in string time format
    '''
    i_hour = int(f_time / 3600)
    i_minute = int((f_time - i_hour * 3600) / 60)
    i_seconds = int((f_time - i_hour * 3600 - i_minute *60 ))
    return '{:02d}:{:02d}:{:02d}'.format(i_hour, i_minute, i_seconds)
```

```python
'''
import zipfile
import csv
import time
f_start = time.time()
s_fname = "data/dol.zip"
fw_out = open('data/ofi_dol.txt', 'w')
fw_out.write('TIME\tOFI\tDELTA_MID\tLOG_RET\n')
archive = zipfile.ZipFile(s_fname, 'r')
f_total = 0.
f_tot_rows = 0.
d_best_price = {'BID': (0., 0.), 'ASK': (0., 0.)}
f_min_time = 10.  # seconds
for i, x in enumerate(archive.infolist()):
    # each file within the zip file (in this case, there is only 1 file)
    f_ofi = 0.
    f_mid = None
    f_next_time = 9 * 3600 + f_min_time
    for idx_row, row in enumerate(csv.DictReader(archive.open(x), delimiter='\t')):
        if idx_row == 0:
            f_first_price = row['Price']
        # no need to deal with trades, because it's already reflected in the bid and ask
        if row['Type'] in ['BID', 'ASK']:
            # converte string para float
            row['Price'] = float(row['Price'].replace(',', '.'))
            row['Size'] = float(row['Size'])
            f_current_time = sum([float(x)*60**(2.-i_aux)  for i_aux, x in enumerate(row['Date'][-8:].split(":"))])
            if f_current_time > f_next_time:
                # print results
                s_time = convert_float_to_time(f_next_time)
                f_change = 0
                f_logrtn = 0.
                if f_mid:
                    f_curent_mid = (d_best_price['ASK'][0] + d_best_price['BID'][0])/2.
                    f_change = int((f_curent_mid - f_mid)/0.5)
                    f_logrtn = np.log((f_curent_mid/f_mid))
                f_mid = (d_best_price['ASK'][0] + d_best_price['BID'][0])/2.
                s_out = '{}\t{}\t{}\t{}\n'.format(s_time, f_ofi, f_change, f_logrtn)
                fw_out.write(s_out)
                # zero value
                f_ofi = 0
                 # prints every 10s
                f_next_time  = (int(f_current_time/f_min_time) + 1)*f_min_time
            elif abs(f_current_time - f_next_time) > 3600:
                # new day
                f_next_time = 9 * 3600
                f_mid = None
                f_ofi = 0
            # compare with last value
            last_best = d_best_price[row['Type']]
            f_e_n = measure_e_n(row, last_best)
            # update last best
            d_best_price[row['Type']] = (row['Price'], row['Size'])
            row['Date'] = row['Date'][-8:]
            f_ofi += f_e_n

print('Time to process: {:0.2f} s'.format(time.time() - f_start))
'''and None
```

Let's take a look at how the data looks like

```python
import pandas as pd
df = pd.read_csv('data/ofi_dol.txt', sep='\t')
df.drop('TIME', axis=1, inplace=True)
df.dropna(inplace=True)
df[['OFI', 'DELTA_MID']].hist(bins=20);

```

It looks like we have some outliers. Let's exclude the most extreme points. To classify these points, I will calculate the value between the 1st and 90th percentiles and check which points are outside 1.5 times this interval in addition to the mentioned percentiles.

```python
import numpy as np
# For each column, find maximum and minimum extreme values
d_unique_idx = dict()
d_iqr = {}
print("Number of data points considered as outliers for the variable\n")
for feature in df.keys():

    # Calculate Q1 (25th percentile of the data) for the given feature
    Q1 = np.percentile(df[feature], 1)

    # Calculate Q3 (75th percentile of the data) for the given feature
    Q3 = np.percentile(df[feature], 90)

    # calcula intervalo interquartil e calcula o tamanho para ser outlier
    d_iqr[feature] = (Q3-Q1)
    step = 1.5*(Q3-Q1)

    # Mostra outliers
    df_out_lier = df[~((df[feature] >= Q1 - step) & (df[feature] <= Q3 + step))]
    i_nfeatures = df_out_lier.shape[0]
    print("{:18s}\t{:18s}".format(feature, str(i_nfeatures)))
    for x in df_out_lier.index:
        if x not in d_unique_idx.keys():
            d_unique_idx[x] = 1
        else:
            d_unique_idx[x] += 1

print("-------------------------")
print("TOTAL: Outliers: {} | Unique Outliers: {}".format(sum(d_unique_idx.values()),
                                                          len(d_unique_idx.keys())))
# filter outliers
print("\nOutliers:")
df.loc[d_unique_idx.keys(), :]
```

    Number of data points considered as outliers for the variable

    OFI               	2
    DELTA_MID         	6
    LOG_RET           	4
    -------------------------
    TOTAL: Outliers: 12 | Unique Outliers: 7

    Outliers:

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>OFI</th>
      <th>DELTA_MID</th>
      <th>LOG_RET</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1881</td>
      <td>-2380.0</td>
      <td>-14.0</td>
      <td>-0.002128</td>
    </tr>
    <tr>
      <td>4782</td>
      <td>1395.0</td>
      <td>7.0</td>
      <td>0.001122</td>
    </tr>
    <tr>
      <td>10</td>
      <td>835.0</td>
      <td>8.0</td>
      <td>0.001204</td>
    </tr>
    <tr>
      <td>11</td>
      <td>655.0</td>
      <td>9.0</td>
      <td>0.001428</td>
    </tr>
    <tr>
      <td>3132</td>
      <td>675.0</td>
      <td>8.0</td>
      <td>0.001302</td>
    </tr>
    <tr>
      <td>3176</td>
      <td>265.0</td>
      <td>35.0</td>
      <td>0.005345</td>
    </tr>
    <tr>
      <td>6036</td>
      <td>1025.0</td>
      <td>12.0</td>
      <td>0.001875</td>
    </tr>
  </tbody>
</table>
</div>

Now, let's plot the scatter plot of the data without the points above

```python
df2 = df.loc[[x for x in df.index if x not in d_unique_idx.keys()]]
```

```python
import seaborn as sns
ax = sns.lmplot(x="OFI", y="DELTA_MID", data=df2, markers=["x"], palette="Set2", height=4, aspect=1.5)
ax.ax.set_title('Relation between $\Delta Mid$ and $OFI$\n', fontsize=15);
```

As expected, there is a clear linear relationship between the volume that entered or left the book and the change in price of the asset, when considered a short period of time. In the next section, I will use this variable to try to improve the result of the VAR model in forecasting the returns.

### 3.3. Fitting the VAR Model to Calculated Data

First, it is necessary to check if the variables are stationary. However, we will do this only visually here.

```python
import matplotlib.pyplot as plt
df_plot = df2.unstack().reset_index()
df_plot.drop('level_1', axis=1, inplace=True)
df_plot.columns = ['Variable', 'Value']
g = sns.FacetGrid(df_plot, col="Variable", sharey=False,  height=4, aspect=1.2)
g = g.map(plt.plot, "Value")
```

The average seems to be constant, since the standard deviation should not be. Also note that all variables have a different scale. Now, let's see what the best VAR order fits into this data (disregarding the return log).

```python
df3 = df2.copy()
```

```python
import var_model.vector_autoregression as var
import numpy as np
import pandas as pd
self = var.VectorAutoregression(pd.DataFrame(df3[['OFI', 'DELTA_MID']]))
self.select_order(18)
```

    Ordem com menor valor para cada Critério:
      Criterion FPE:  		Ord. 16
      Criterion AIC:  		Ord. 16
      Criterion HQ:  		Ord. 6
      Criterion SC(BIC):  		Ord. 1

              FPE     AIC      HQ  SC(BIC)
    Ordem
    1      1.6930  9.2636  9.2651   9.2680
    2      1.6927  9.2636  9.2666   9.2723
    3      1.6818  9.2573  9.2618   9.2703
    4      1.6766  9.2544  9.2604   9.2717
    5      1.6697  9.2504  9.2579   9.2720
    6      1.6666  9.2487  9.2577   9.2747
    7      1.6667  9.2489  9.2594   9.2792
    8      1.6672  9.2494  9.2614   9.2840
    9      1.6675  9.2497  9.2633   9.2887
    10     1.6679  9.2501  9.2651   9.2934
    11     1.6671  9.2498  9.2663   9.2975
    12     1.6669  9.2499  9.2679   9.3018
    13     1.6647  9.2487  9.2682   9.3050
    14     1.6669  9.2502  9.2712   9.3108
    15     1.6634  9.2482  9.2708   9.3132
    16     1.6593  9.2459  9.2700   9.3153
    17     1.6623  9.2479  9.2734   9.3216
    18     1.6625  9.2482  9.2752   9.3262

The criteria indicated different orders. Let's see how the projections look with a model of order 16, initially.

```python
def measure_on_data(i_var_order, l_col):
    var_6 = var.VectorAutoregression(df3[l_col])
    var_6.fit(i_var_order)
    l_rtn = []
    # f_val = float(f_first_price.replace(',', '.'))
    f_val = 0
    df4 = df3[l_col].reset_index(drop=True)
    l_label = df4.columns
    for i_idx in range(i_var_order, df4.shape[0]):
        na_y = df4.loc[(i_idx-i_var_order):i_idx-1, :].values
        na_rtn, na_max, na_min= var_6.forecast(na_y, 1)
        na_yr = df4.loc[i_idx, :]

        d_aux = {'ID': i_idx}
        for x in range(len(l_label)):
            d_aux[l_label[x] + '_OBSV'] = na_yr[x]
            d_aux[l_label[x] + '_FORC'] = na_rtn[x]
            d_aux[l_label[x] + '_MAX'] = na_max[x]
            d_aux[l_label[x] + '_MIN'] = na_min[x]

        l_rtn.append(d_aux)

    df_test = pd.DataFrame(l_rtn)
    df_test.index = df_test['ID']
    df_test.drop('ID', axis=1, inplace=True)

    return df_test
```

```python
df_test = measure_on_data(16, l_col=['OFI'])
ax = df_test.plot(alpha=0.7)
ax.set_title('Comparison between true value of $OFI$ and forecast \n', fontsize=15);
```

Note that the forecasted value ($ OFI \_FORC $) was far from what was observed. So, instead of looking at the forecasted value, let's focus on the sign of the values and check if the forecasted value pointed to the same side as the realized one. Below I will plot the percentage of times that the model hit the market side, adjusting several VARs of different orders with different combinations of variables. The objective is always to predict the first variable of each line, according to the legend.

```python
ax = None
for l_col in [['LOG_RET'], ['OFI'], ['DELTA_MID'], ['LOG_RET', 'OFI'], ['DELTA_MID', 'OFI']]:
    d_rtn = {}
    l_orders_to_test = [1, 6, 16, 30, 90]  # 10 seg, 40 seg, 2 min e 40 seg, 5 min, 15 min
    for i_ord in l_orders_to_test:
        df_test = measure_on_data(i_ord, l_col=l_col)
        d_rtn[i_ord] = ((df_test[l_col[0] + '_FORC'] / df_test[l_col[0] + '_OBSV'])>=0).sum() * 1. / df_test.shape[0]

    if not ax:
        ax = pd.Series(d_rtn).plot(ax=ax, label = str(l_col)[1:-1])
    else:
        pd.Series(d_rtn).plot(label = str(l_col)[1:-1])
ax.set_title(u'Observation and Forecast', fontsize=16)
ax.legend(loc='lower right')
ax.set_xlabel('Model Order')
ax.set_ylabel('Percentage');
```

When a VAR model was fitted to only one variable, the one that did best in low orders was the one fitted to predict the OFI, making the next observation of this variable a little more than 50% of the time. However, when we increase the order of the model, the models fitted to the return log and the delta-mid outperform the model fitted only to the OFI. When using more than one variable, the models using the OFI and the mid-price or the OFI and the log-return showed similar performance in models of order up to 50, coming close to hitting the market direction almost 55% of the time . However, in higher orders, the model that used the OFI and the delta-mid was the one that did better, surpassing 55% of accuracy.

## 4. Conclusion

Tests with the VAR model using the proposed variables for high frequency data indicate that it is not suitable if accurate * forecasts * are required. However, when considering the direction of the signal given by the model, the results suggest that it can be a useful tool in the context of * trading *.

## 5. Last Considerations

It would be interesting to perform a back-test with the created models and try to add more variables to analyze if the forecasting will be improved.

## References

1. Ruey S. Tsay. *Analysis of Financial Time Series*. Wiley 2010, Financial Engineering. [*link*](https://www.wiley.com/en-us/Analysis+of+Financial+Time+Series%2C+3rd+Edition-p-9780470414354)
2. H. Lütkepohl.  *New Introduction to Multiple Time Series Analysis*. Springer-Verlag, 2005. [*link*](http://www.springer.com/br/book/9783540401728)
3. Heij, C. and de Boer, P. and Franses, P.H. and Kloek, T. and van Dijk, H.K. and Rotterdam.  *Econometric Methods with Applications in Business and Economics*. OUP Oxford, 2004. [*link*](https://books.google.com.br/books?id=hp4vQZZHfbUC)
4. Cont, R. and Kukanov, A. and Stoikov.  *The Price Impact of Order Book Events*. Journal Of Financial Econometrics, 2014. [*link*](http://ssrn.com/abstract=1712822)


## Simple qubit rotation with TFQ

In this jupyter file we define a variational quantum circuit $V(\theta)$ that rotates an initial state $|0000\rangle$ into a target state with equal superposition $\frac{1}{\sqrt{|\Sigma|}}\sum_{\sigma_i} | \sigma_i \rangle$. The aim is that  $\frac{1}{\sqrt{|\Sigma|}}\sum_{\sigma_i} \langle \sigma_i | V(\theta) | 0000\rangle = 1$.

```python
import tensorflow as tf
import tensorflow_quantum as tfq
from tensorflow.keras import Model

import cirq
import sympy
import numpy as np

# visualization tools
%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit
```

## Define the circuit with Cirq

To measure the overlap with the target state, we can simply measure the operator $M = 1/4*(X_1 + X_2 + X_3 + X_4)$
and minimize the loss $1-\langle M \rangle$.

We implement this mesurement operator as `op`

```python
def generate_circuit(nr_of_qubits, layers):
    qubits = cirq.GridQubit.rect(1, nr_of_qubits) # Define qubit grid. In this case
    nr_parameters = 3*nr_of_qubits*layers # 3 params for each qubit and layer

    symb = sympy.symbols('theta0:'+str(nr_parameters))
    symbols = np.array(symb)
    symbols = symbols.reshape(layers, nr_of_qubits, 3)
    circuit = cirq.Circuit()

    for l in range(layers):
        # Add a series of single qubit rotations.
        for i, qubit in enumerate(qubits):
            circuit += cirq.rz(symbols[l][i][0])(qubit)
            circuit += cirq.rx(symbols[l][i][1])(qubit)
            circuit += cirq.rz(symbols[l][i][2])(qubit)

        circuit += cirq.CZ(qubits[0], qubits[1])
        circuit += cirq.CZ(qubits[2], qubits[3])
        circuit += cirq.CZ(qubits[1], qubits[2])

    op = 1/4*(cirq.X(qubits[0]) + cirq.X(qubits[1]) + cirq.X(qubits[2]) + cirq.X(qubits[3]))
    return circuit, op, list(symb)

nr_of_qubits = 4
layers = 2
tf_circuit, op, (symbols) = generate_circuit(nr_of_qubits, layers)
SVGCircuit(tf_circuit)
```

## Method without keras

### Define a classical model that outputs the circuit parameters

These parameters will be trained. This is the basic structure this model needs to have. But we could `call()` any function or even whole nueral networks that return these parameters.

```python
class circuit_params(Model):
    def __init__(self, W = np.random.normal(2)):
        super(circuit_params, self).__init__()

        self.W = tf.Variable(W)

    @tf.function
    def __call__(self):
        return self.W

model = circuit_params(W = np.random.rand(nr_of_qubits*layers*3))
model.trainable_variables
```

    [<tf.Variable 'Variable:0' shape=(24,) dtype=float64, numpy=
     array([0.4139659 , 0.34282132, 0.87948004, 0.65461008, 0.32933811,
            0.80638918, 0.07704536, 0.03909048, 0.41425862, 0.14154555,
            0.72498658, 0.19544217, 0.87069088, 0.46945953, 0.05214077,
            0.87803274, 0.0761321 , 0.5159643 , 0.50242537, 0.1400908 ,
            0.05550696, 0.10629389, 0.16118076, 0.63178469])>]

### Make a test run to get the expectation

For a randomly initialized model we get the following expectation value

```python
optimizer = tf.keras.optimizers.SGD(lr=0.5)

params = model()

out = tfq.layers.Expectation()(tf_circuit,
                         symbol_names=symbols,
                         symbol_values=[params],
                         operators=1-op)

out
```

    <tf.Tensor: shape=(1, 1), dtype=float32, numpy=array([[0.6974215]], dtype=float32)>

### Define training routine

For the training we call the model params and forward them to the TFQ layer for the expectation value. This calculations is intergrable with TFQ and we get the gradient and can update the parameters with the optimizer.

```python
@tf.function
def train_step():
    with tf.GradientTape() as tape:
        # training=True is only needed if there are layers with different
        # behavior during training versus inference (e.g. Dropout).
        params = model()
        out = tfq.layers.Expectation()(tf_circuit,
                         symbol_names=symbols,
                         symbol_values=[params],
                         operators=1-op)

    gradients = tape.gradient(out, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
```

#### Run the training

```python
for i in range(100):
    train_step()
```

#### Check training success

```python
params = model()
out = tfq.layers.Expectation()(tf_circuit,
                 symbol_names=symbols,
                 symbol_values=[params],
                 operators=1-op)

out
```

    <tf.Tensor: shape=(1, 1), dtype=float32, numpy=array([[2.2828579e-05]], dtype=float32)>

## Implement Keras method

This way is much shorter but it might give us less access to all the details of the circuit. And it is much slower than the other version.

### Transform circuit into TF layer

```python
outputs = tfq.layers.PQC(tf_circuit,         # Circuit to be transformed into tf layer
                         1-op,               # Output of the layer
                         repetitions=1024,
                         differentiator=tfq.differentiators.ParameterShift())
```

### Define a model with a random input and the output of the quantum layer

```python
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(), dtype=tf.string),
    outputs
])
```

In order to compile and fit the model, one needs to define a loss function (the quantity to optimize) and an optimizer. We want to optimize the expectation value of $M$, this is, the output of the model. Tensorflow needs this, however, as a function ``f(real_values,predictions)``

```python
loss = lambda real, pred: pred    # Forget about any "real" values, just consider as loss the model's output

model.compile(loss=loss,
              optimizer=tf.keras.optimizers.SGD(learning_rate=0.4)) # Same optimizer as the Pennylane case
```

Finally, the ``fit`` function is designed for classification problems, and thus it needs of some inputs and corresponding "true" outputs. Our variational quantum circuit has none of these, so we just insert empty attributes: the input will be an empty quantum circuit, which corresponds to setting the initial state to $|0000\rangle$; the output will be an empty array (you can choose anything you want, since the loss function will discard whatever you set)

```python
dummy_input  = tfq.convert_to_tensor([cirq.Circuit()])
dummy_output = np.array([[]])
```

```python
steps = 100
model.fit(dummy_input, dummy_output, epochs=steps)
```

    Train on 1 samples
    Epoch 1/100
    WARNING:tensorflow:AutoGraph could not transform <bound method ParameterShift.differentiate_sampled of <tensorflow.python.eager.function.TfMethodTarget object at 0x1377c35d0>> and will run it as-is.
    Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
    Cause: unexpected indent (<unknown>, line 77)
    WARNING: AutoGraph could not transform <bound method ParameterShift.differentiate_sampled of <tensorflow.python.eager.function.TfMethodTarget object at 0x1377c35d0>> and will run it as-is.
    Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
    Cause: unexpected indent (<unknown>, line 77)
    1/1 [==============================] - 4s 4s/sample - loss: 1.0786
    Epoch 2/100
    1/1 [==============================] - 4s 4s/sample - loss: 0.9565
    Epoch 3/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.8589
    Epoch 4/100
    1/1 [==============================] - 4s 4s/sample - loss: 0.7798
    Epoch 5/100
    1/1 [==============================] - 4s 4s/sample - loss: 0.7183
    Epoch 6/100

```python

```


# LWR Traffic flow with varying speed limit

```python
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib as mpl
mpl.rcParams['font.size'] = 8
figsize =(8,4)
mpl.rcParams['figure.figsize'] = figsize

import matplotlib.pyplot as plt
import numpy as np
from ipywidgets import interact
from ipywidgets import widgets, FloatSlider
from utils import riemann_tools
from exact_solvers import traffic_variable_speed
```

## The model

In this chapter we again consider the LWR traffic model, but now we introduce a speed limit $v$:

\begin{align} \label{LWR_vc:conslaw}
\rho_t + (v(x) \rho(1-\rho))_x = 0.
\end{align}
and we consider the Riemann problem with different speed limits to the left and right:

$$
v(x) = \begin{cases} v_l & x<0 \\ v_r & x>0. \end{cases}
$$

In [the earlier LWR chapter](Traffic_flow.ipynb), the speed limit was set to unity everywhere.  The variable-speed-limit case has been considered, for instance, in (Mochon, 1987) and in Chapter 16 of (LeVeque, 2002).  From a physical point of view, we might also imagine that $u_\text{max}$ varies because of differing road conditions -- for instance, if part of the road is wet or foggy.

Correspondingly, we have a flux function that is different on either side of $x=0$.  Let $f(\rho,v) = v \rho (1-\rho)$ and define

\begin{align}
    f_l & = f(\rho_l,v_l) \\
    f_r & = f(\rho_r,v_r).
\end{align}

The presence of a discontinuous flux function in the Riemann problem can introduce new difficulties, some of which will be illustrated here.  For a more detailed discussion of other systems with discontinuous flux, see (Burger et. al., 2008) and other papers in that special issue.

Continuing with the problem at hand, if, say, $v_l=1$ and $v_r=2$ then the two flux functions look like this:

```python
rho = np.linspace(0,1)
f_l = rho*(1-rho); f_r = 2*rho*(1-rho)
plt.plot(rho,f_l,'-c',rho,f_r,'-g'); plt.xlim(0,1); plt.ylim(0,0.6);
plt.xlabel(r'$\rho$'); plt.ylabel(r'$f(\rho)$')
plt.legend(['$f_l$','$f_r$']);
```

## Admissible regions
We can view the solution of the Riemann problem as the task of finding a physically admissible path connecting the left and right states in the $\rho-f(\rho)$ plane.  The structure of the Riemann solution can be fairly complicated, but we can simplify things by recognizing a few "rules" that must govern the connecting path.  Two quick observations will rule out half of the available intermediate states:

1. Since the characteristic speed is positive for $\rho<1/2$, **we cannot reach any intermediate state with $\rho<1/2$ on the $f_l$ curve** (since states on the $f_l$ curve must appear to the left of $x=0$).
2. Similarly, since the characteristic speed is negative for $\rho>1/2$, **we cannot reach any intermediate state with $\rho>1/2$ on the $f_r$ curve**.

To remind us of these facts, we'll plot the prohibited parts of the curves with dashed lines.  It is important to note that the initial states $\rho_l$ and $\rho_r$ **can** inhabit these regions.  However, if an initial state is on the inadmissible portion of the curve, then we cannot connect a rarefaction to it, since nearby states are inadmissible.  We must jump to it through a rarefaction or (as discussed below) a stationary jump.

```python
f_l = rho*(1-rho); f_r = 2*rho*(1-rho)
plt.plot(rho[26:],f_l[26:],'-c',rho[:26],f_r[:26],'-g');
plt.xlim(0,1); plt.ylim(0,0.6);
plt.plot(rho[:26],f_l[:26],'--k',rho[26:],f_r[26:],'--k',alpha=0.5);
plt.xlim(0,1); plt.ylim(0,0.6);
plt.xlabel(r'$\rho$'); plt.ylabel(r'$f(\rho)$')
plt.legend(['$f_l$','$f_r$']);
```

## Shocks and rarefactions
We can also determine in advance what kind of waves can be connected to $\rho_l$ or $\rho_r$ within the admissible regions:

1.  Any intermediate state on the $f_l$ curve with density greater than $\rho_l$ must be connected to $\rho_l$ by a shock.  Conversely, any intermediate state with density less than $\rho_l$ must be connected to $\rho_l$ by a rarefaction.
2.  Similarly, any intermediate state on the $f_r$ curve with density greater than $\rho_r$ must be connected to $\rho_r$ by a rarefaction, and any intermediate state with density less than $\rho_r$ must be connected to $\rho_r$ by a shock.

These conditions follow directly from the Lax entropy condition.  To remind us of these conditions, we'll plot in red the portions of the admissible curves that can be connected to the left or right state by a shock, and in blue those that can be connected by a rarefaction.  Here's an example of what that looks like.

```python
rho_l = 0.6; rho_r = 0.6
v_l = 1.0; v_r = 0.8
traffic_variable_speed.phase_plane_plot(rho_l,rho_r,
                                        v_l,v_r,connect=False)
```

## The stationary wave

So far we've only discussed how to connect the left and right states to other states on the same flux curve, but to solve the Riemann problem we must connect points on the two different flux curves to each other.

As discussed at length in [the earlier traffic flow chapter](Traffic_flow.ipynb), the flux of vehicles must be continuous everywhere -- and in particular, at $x=0$:
\begin{align} \label{LWR_vc:fluxcont}
f(\rho(0^-,t),v_l)) = f(\rho(0^+,t),v_r)) \equiv f^*.
\end{align}

If the speed limit changes abruptly at $x=0$, then (in order to have a continuous flux) the density must be discontinuous there.  Since there are two possible values of $\rho$ corresponding to any given flux, we must carefully determine the intermediate states.  We can think of the discontinuity at $x=0$ as an additional wave with speed zero.

It may seem strange that the Riemann solution includes two waves when we are dealing with a scalar PDE.  One way to understand this is by adding a trivial evolution equation for the speed limit (which is now artificially considered a function of $t$):

\begin{align} \label{LWR_vc:2by2}
    \rho_t + (v(x,t) \rho(1-\rho))_x & = 0 \\
    v_t & = 0.
\end{align}

The flux jacobian for this system is a $2\times 2$ matrix with one eigenvalue equal to the characteristic speed $f'(\rho)$ and the other equal to zero.  Thus we expect its solution to include a wave with speed zero.  We will not investigate the hyperbolic structure of this $2\times 2$ system further here; instead the flux continuity condition \eqref{LWR_vc:fluxcont} together with an admissibility condition (discussed below) will be enough to enable us to construct solutions to the Riemann problem based on the scalar equation form \eqref{LWR_vc:conslaw}.

Since the flux is continuous at $x=0$, the stationary jump there corresponds to a horizontal line in the $\rho-f(\rho)$ plane plotted above.  This jump connects a point on one flux curve to a point on the other.  Two things are immediately clear:

1. The value of $f^*$ must not lie fully above either of the flux curves; since the maximum flux is $v/4$, this means that $f^* \le \min(v_l,v_r)/4$.  If this condition were violated, it would be impossible to connect the two flux curves with a horizontal line at flux value $f^*$.
2. For a given $f^*$ satisfying condition 1, there are always **two** values of $\rho$ on each curve with flux $f^*$.  This means that if (say) we have determined the value $\rho(x=0^-,t)$, there are two candidate values of $\rho(x=0^+,t)$ -- one with positive characteristic velocity, and one with negative characteristic velocity.

How can we determine which state to jump to?  We will apply an admissibility condition similar to the Lax entropy condition that allowed us to distinguish whether a wave should be a shock or rarefaction.  Recall that:

1. For **shock waves**, characteristics near that of the shock must be impinging from both sides: $c_l>s>c_r$.
2. For **rarefaction waves**, at any characteristic within the rarefaction, nearby characteristics must be **diverging** from both sides: $c_l<s<c_r$.
3. For **contact waves**, nearby characteristics must be parallel to the contact wave on both sides: $c_l=s=c_r$.

The stationary wave corresponds to the zero eigenvalue of the flux Jacobian of \eqref{LWR_vc:2by2}; evidently
all characteristics in this family have speed zero and are thus parallel to the stationary jump.  We cannot use those characteristics to distinguish the admissibility of the jump.  Instead, we impose the following admissibility criteria, related to the original characteristic speed of \eqref{LWR_vc:conslaw}:

1. The characteristic velocity does not change sign across the wave at $x=0$ (in a weak sense). That is, $f_l'(\rho(x=0^-,t)) f_r'(\rho(x=0^+,t)) \ge 0$.
2. If the characteristic velocity vanishes on one side of the stationary wave, the value on the other side must lie on the admissible portion of the flux curve.

Since the sonic point $\rho=1/2$ plays such a crucial role, we will say a section of road is *congested* if $\rho$ is greater than $1/2$ there.

Finally, let us formalize one thing that we observed in the previous notebook:

>In the LWR system, shocks always carry an increase in density (from left to right), while rarefactions always carry a decrease in density.

Why is this?  The Lax entropy condition for a shock moving at speed $s$ tells us

$$f'(\rho_l) > s > f'(\rho_r).$$

In partcular, $f'(\rho_l) > f'(\rho_r)$, which for LWR means $1-2\rho_l > 1-2\rho_r$, or $\rho_r > \rho_l$.  This guarantees that characteristics are converging in the neighborhood of the shock.  At a rarefaction, characteristics must be diverging, so we must have the opposite inequality: $\rho_r < \rho_l$.

Now let's look at some specific cases and see how to solve the Riemann problem by connecting the left and right states.  We'll start with the states already plotted above.

## Case I: speed limit decrease with $\rho_l > \rho_r$

```python
rho_l = 0.6; rho_r = 0.6
v_l = 1.0; v_r = 0.8
traffic_variable_speed.phase_plane_plot(rho_l,rho_r,
                                        v_l,v_r,connect=False)
```

From the rules determined above, we know that we can connect $\rho_r$ to an intermediate state only by the stationary jump or by a shock.  If it is connected by a shock, the intermediate state must lie on the red part of the $f_r$ curve.  But then the stationary jump would need to either cross the midline (not allowed) or connect to the inadmissible left portion of the $f_l$ curve (also not allowed).  Hence $\rho_r$ must be connected by the stationary wave to the admissible (red) portion of the $f_l$ curve.  It is then clear that $\rho_l$ must be connected to the intermediate state by a shock.

Here is what the solution looks like:

```python
def c(rho, xi, v):
    return v*(1.-2*rho)

def make_plot_function(rho_l,rho_r,v_l,v_r,connect=True):
    states, speeds, reval, wave_types = \
        traffic_variable_speed.exact_riemann_solution(rho_l,rho_r,
                                                      v_l,v_r)
    def plot_function(t):
        ax = riemann_tools.plot_riemann(states,speeds,
                          reval,wave_types,t=t,t_pointer=0,
                          extra_axes=True,variable_names=['Density']);
        riemann_tools.plot_characteristics(reval,c,(v_l,v_r),ax[0],
                          extra_lines=[[[0,0],[0,1]]])
        traffic_variable_speed.phase_plane_plot(rho_l,rho_r,v_l,v_r,
                          axes=ax[2],connect=connect)
        plt.show()
    return plot_function

def plot_riemann_traffic_vc(rho_l,rho_r,v_l,v_r,connect=True):
    plot_function = make_plot_function(rho_l,rho_r,v_l,v_r,connect)
    interact(plot_function,
             t=widgets.FloatSlider(value=0.5,min=0,max=.9))
```

```python
plot_riemann_traffic_vc(rho_l,rho_r,v_l,v_r);
```

The $\rho-f$ plane is shown in the rightmost plot above.  The Riemann solution is indicated with a thick line; as usual we indicate shocks in red and rarefactions in blue; the stationary jump at $x=0$ is plotted in black.  Notice that the jump at the interface corresponds to a horizontal line, since the flux is the same on both sides of the interface.  Notice also that the jump does not cross the dashed line at $\rho=1/2$.  The solution shown satisfies all of the admissibility criteria laid out above; indeed, it is the only solution that does so.

## Case 2: speed limit decrease with $\rho_l < \rho_r$
In the problem above, it was impossible to connect $\rho_l$ directly to the $f_r$ curve since $\rho_l$ was greater than $v_r/4$.  Here is a problem with the same speed limits but a smaller value of $\rho_l$.

```python
rho_l = 0.9; rho_r = 0.6
v_l = 1.0; v_r = 0.8
traffic_variable_speed.phase_plane_plot(rho_l,rho_r,
                                        v_l,v_r,connect=False)
```

How can we connect these states?  By the same reasoning used in the previous problem, $\rho_r$ must be connected to the $f_l$ curve by the stationary wave.  Then it must be that $\rho_l$ is connected to the intermediate state by a rarefaction.

```python
plot_riemann_traffic_vc(rho_l,rho_r,v_l,v_r);
```

Again, observe how the waves involved satisfy all of the required conditions.  No other set of waves does so.

## Case 3: Transonic rarefaction
So far we have seen solutions with a single shock or rarefaction; they are similar to solutions of the Riemann problem for the LWR model with fixed speed limit (see [this earlier example](Traffic_flow.ipynb#Example:-Traffic-jam) and [this earlier example](Traffic_flow.ipynb#Example:-green-light), except that they also include a stationary jump.  More interesting things can happen if we have a rarefaction in which the characteristic speed passes through zero (i.e., a *transonic rarefaction*).  In order to connect a rarefaction to both states in the variable-speed-limit setting, the densities $\rho_l$ and $\rho_r$ bounding the rarefaction wave must both lie on the admissible portions of the curve.  Thus we must have $\rho_r \le 1/2 \le \rho_l$; this is the same condition that was required for a transonic rarefaction in the basic LWR model.

However, in the variable-speed-limit case there are additional conditions that must be satisfied.  Namely, the left and right fluxes must not be so great that we cannot balance them with the appropriate flux on the other side; i.e. we require

\begin{align}
f_l & \le v_r/4 \label{vcLWR:sufficientA} \\
f_r & \le v_l/4.\label{vcLWR:sufficientB}
\end{align}
This guarantees that neither the left nor right state lies above the entire (right or left) flux curve in the phase plane.

### Transonic rarefaction with $v_l > v_r$
Here's an example of a Riemann problem that leads to a transonic rarefaction:

```python
rho_l = 0.9; rho_r = 0.2
v_l = 1.0; v_r = 0.8
traffic_variable_speed.phase_plane_plot(rho_l,rho_r,
                                        v_l,v_r,connect=False)
```

Exactly how will we connect these states?  In order to obtain an admissible stationary jump, there is only one possibility: we must connect $\rho_r$ to the sonic point via a rarefaction, and then connect to an admissible point on the right side of the $f_l$ curve via the stationary jump.  Finally, the resulting state is connected to $\rho_l$ by another rarefaction.

```python
plot_riemann_traffic_vc(rho_l,rho_r,v_l,v_r);
```

Because we have a stationary jump in $\rho$ in order to maintain flux continuity at the interface, it is impossible that the characteristic velocity approach zero on *both* sides of the stationary jump.  Thus the transonic rarefaction is broken into two rarefactions, with an intermediate constant state lying either just to the left or just to the right of $x=0$.  Where this state lies depends on whether the speed limit increases or decreases at the interface; in the example above this "shoulder" appears on the right.  Next is an example in which it appears on the left.

### Transonic rarefaction with $v_r > v_l$

```python
rho_l = 0.9; rho_r = 0.2
v_l = 1.0; v_r = 1.2
plot_riemann_traffic_vc(rho_l,rho_r,v_l,v_r);
```

## Case 4: Transonic rarefaction-shock
In the last two examples above we required the conditions \eqref{vcLWR:sufficientA}-\eqref{vcLWR:sufficientB}, ensuring that neither initial state lies fully above the other flux curve.  What if one of these conditions is violated?

Here is a case where $\rho_l$ lies in the admissible region while $\rho_r$ lies above the value $\max(f_l)=v_l/4$.

```python
rho_l = 0.7; rho_r = 0.6
v_l = 1.0; v_r = 2.
traffic_variable_speed.phase_plane_plot(rho_l,rho_r,
                                        v_l,v_r,connect=False)
```

Now $\rho_r$ cannot be connected to anything via a rarefaction, and cannot be connected to a state on the $f_l$ curve by a stationary jump (i.e., a horizontal line).  The only possibility then is to connect $\rho_r$ to the admissible (left) part of the $f_r$ curve, evidently by a shock.  But then how can we connect the intermediate state back to $\rho_l$, which lies to the left of the midline?  This is only possible by connecting from the $f_r$ curve to the $f_l$ curve at the sonic point $\rho=1/2$.  Since this connection corresponds to the stationary jump and must be a horizontal line, this condition determines the intermediate state on the $f_r$ curve.  Then $\rho_l$ must be connected to the sonic point by a rarefaction.

```python
plot_riemann_traffic_vc(rho_l,rho_r,v_l,v_r);
```

Physically, this corresponds to the situation in which the downstream flux is so high that the influx of cars from upstream cannot possibly keep up.  The density just downstream of the jump must be lower than that just upstream.  Thus we see a left-going rarefaction as the upstream congestion decreases since downstream traffic flux is greater; a stationary jump at the interface, where cars instantaneously accelerate due to the higher speed limit; and a right-going shock carrying a drop in density (from right to left) as the road clears out since there is less traffic arriving from upstream.

We are solving a scalar problem, and yet we see three waves!  Even writing it artificially as a system of two equations (as we did above) doesn't account for this.  What's going on?

In fact, this case is not so different from that of the transonic rarefaction; observe that in the solution above we have half of a transonic rarefaction; i.e. a rarefaction in which on edge coincides with the $x=0$ characteristic.  However, the stationary jump connects this rarefaction to a state with characteristic velocity greater than that of $\rho_r$.  Thus it's impossible to impose a rarefaction on the right side, and the other half of the transonic rarefaction must instead be a shock.  We can think of this rarefaction-shock pair as a single composite wave that might be called a *transonic rarefaction-shock*.

In the phase plane we see that the rarefaction raises the density as far as it can go on the lower flux curve.  In other words, the flux from the left at the interface is as high as it can possibly be.  Since the characteristic speed just to the left of $x=0$ approaches zero, the condition on the stationary wave that we introduced in the introduction allows a jump to either side of the $f_r$ curve; however, only the state to the left is admissible.  Observe that if the density increased at $x=0$ (by jumping to the right side), then the resulting intermediate state would have to be connected to $\rho_r$ by a (*right-going*) rarefaction in which all characteristics are *left-going*, which makes no sense.

Mathematically, the transonic rarefactions and transonic rarefaction-shock that we have just examined are examples of *resonant waves*.  In a strictly hyperbolic system, the characteristic speeds can always be ordered strictly so that

$$
\lambda_1 < \lambda_2 < \cdots < \lambda_m.
$$

However, if two characteristic speeds coincide, then the system is non-strictly hyperbolic.  Considering the $2 \times 2$ form of the current model \eqref{LWR_vc:2by2}, we see that strict hyperbolicity is lost when the characteristic speed $v \cdot (1-2\rho)$ vanishes.  The resulting resonant interaction between the characteristic families leads to different behavior than that of a strictly hyperbolic system; for instance, the appearance of additional waves in the Riemann problem here and (in some cases, like the last solution above) an increase in the total variation of the solution.  See for example (Isaacson & Temple, 1992) for a discussion of such problems.

## A fictitious solution
In the previous example, you might wonder why there is an intermediate state with $c>0$ when both $\rho_r$ and $\rho_l$ have $c<0$.  In order to connect these two states without such an intermediate state, we can imagine imposing only a jump at the interface and a right-going shock or rarefaction.  Here is what we get by imposing a shock:

```python
states = np.array([[rho_l,0.88,rho_r]])
speeds = np.array([0,0.5])
def reval(xi):
    rho = np.ones((1,len(xi)))
    return rho
wave_types = ['contact','shock']
fig, ax = plt.subplots(1,2,figsize=figsize)
traffic_variable_speed.phase_plane_plot(rho_l,rho_r,v_l,v_r,states,
                       speeds,reval,wave_types,axes=ax[1],show=False)
riemann_tools.plot_waves(states,speeds,reval,wave_types,ax=ax[0],
                         t_pointer=False,t=0)
riemann_tools.plot_characteristics(reval,c,(v_l,v_r),ax[0])
```

As we can see, the shock is unphysical.  The characteristics to the left of the shock are emerging from it, rather than impinging on it; this violates the entropy condition.  It's also easy to see why we can't replace that shock with a rarefaction: the rarefaction would be right-going, but all the neighboring characteristics are left-going!  Finally, this solution involves connecting to an intermediate inadmissible state, which (according to our reasoning in [the beginning of the chapter](Traffic_variable_speed.ipynb#Admissible-regions)) is incorrect.

## Case 5: Transonic rarefaction-shock with speed limit decrease
Finally, let us consider a situation similar to Case 4, but with the situation reversed.

```python
rho_l = 0.7; rho_r = 0.1
v_l = 1.0; v_r = 0.4
traffic_variable_speed.phase_plane_plot(rho_l,rho_r,
                                        v_l,v_r,connect=False)
```

Through similar reasoning, it is evident that these states must be connected through a right-going rarefaction and a left-going shock.

```python
plot_riemann_traffic_vc(rho_l,rho_r,v_l,v_r);
```

Physically in this case the flux upstream of the jump is so great that the right section of road (with lower speed limit) cannot let it all through fast enough.  Since there is then a high density of cars at $x=0$, we might expect a rarefaction as they spread out into the uncongested section of road.  That is indeed what happens.

This may seem counterintuitive at first, but is a familiar experience.  For instance, when a highway narrows to fewer lanes, it often happens that a traffic jam forms in the region just before where it narrows.  Just as one reaches the point where it narrows, the jam clears and one is able to accelerate because the next section of road is less congested (even though it is narrower).

## Other solutions
We've now considered all the essential types of solutions.  *You can continue to experiment with this system using the interactive setup below.  See if you can determine the correct path in the $\rho-f(\rho)$ plane before revealing the result (using the checkbox).*

```python
def plot_all(rho_l,rho_r,v_l,v_r,connect=False):
    states, speeds, reval, wave_types = \
        traffic_variable_speed.exact_riemann_solution(rho_l,rho_r,v_l,v_r)
    ax = riemann_tools.plot_riemann(states,speeds,reval,wave_types,
                       t=0.5,extra_axes=True,variable_names=[r'$\rho$']);
    riemann_tools.plot_characteristics(reval,c,(v_l,v_r),ax[0],
                                       extra_lines=[[[0,0],[0,1]]])
    ax[1].set_ylim(0,1)
    traffic_variable_speed.phase_plane_plot(rho_l,rho_r,v_l,v_r,
                                            axes=ax[2],connect=connect)
    plt.show()

interact(plot_all,
         rho_l=FloatSlider(min=0.,max=1.,step=0.01,value=0.4,
                           description=r'$\rho_l$'),
         rho_r=FloatSlider(min=0.,max=1.,step=0.01,value=0.7,
                           description=r'$\rho_r$'),
         v_l=FloatSlider(min=0.1,max=2.,value=1.),
         v_r=FloatSlider(min=0.1,max=2.,value=0.4),
         connect=widgets.Checkbox(description='Connect states')
         );
```

## Approximate solvers

In order to define a numerical solver, we need to compute fluctuations (for the first-order solver) as well as waves and speeds (for the second-order corrections).

### Fluctuations
It is straightforward to compute the fluctuations by first determining the flux at the interface (i.e., the Godunov flux) $f^*$.  We can then determine the fluctuations via

\begin{align}
    {\mathcal A}^-\Delta Q & = f^* - f_l \\
    {\mathcal A}^+\Delta Q & = f_r - f^*.
\end{align}

In fact, $f^*$ is always equal to one of $\{f_l, f_r, v_l/4, v_r/4\}$.  We can use the criteria discussed above to determine which of these possibilities is correct.

### Second-order corrections
The fluctuations above are the exact fluctuations from the true solution of the Riemann problem.  We could take a similar approach and include the exact wave(s) appearing in the Riemann solution in order to form second-order corrections.  This may include one or two waves (we need not include the stationary jump at the interface since it doesn't modify the value of the solution in either cell).  However, a simpler approach discussed in (LeVeque, 2002) seems to work well.  We use a single wave with strength $\rho_r-\rho_l$ and speed equal to the Rankine-Hugoniot average speed $(c_r + c_l)/2$.

```python
from clawpack import pyclaw
from clawpack import riemann
```

```python
def test_solver(rho_l, rho_r, v_l, v_r, riemann_solver):
    solver = pyclaw.ClawSolver1D(riemann_solver)

    solver.bc_lower[0] = pyclaw.BC.extrap
    solver.bc_upper[0] = pyclaw.BC.extrap
    solver.aux_bc_lower[0] = pyclaw.BC.extrap
    solver.aux_bc_upper[0] = pyclaw.BC.extrap

    x = pyclaw.Dimension(-1.0,1.0,50,name='x')
    domain = pyclaw.Domain(x)
    num_aux = 1
    state = pyclaw.State(domain,solver.num_eqn,num_aux)

    grid = state.grid
    xc=grid.p_centers[0]

    state.q[0,:] = rho_l*(xc<0) + rho_r*(xc>=0.)
    state.aux[0,:] = v_l*(xc<0) + v_r*(xc>=0.)  # Speed limit

    claw = pyclaw.Controller()
    claw.tfinal = 1.0
    claw.solution = pyclaw.Solution(state,domain)
    claw.solver = solver
    claw.keep_copy = True
    claw.verbosity = 0

    claw.run()
    return xc, claw.frames
```

```python
rho_l = 0.7; rho_r = 0.5
v_l = 1.0; v_r = 1.5
t = 0.5

states, speeds, reval, wave_types = \
    traffic_variable_speed.exact_riemann_solution(rho_l,rho_r,
                                                  v_l,v_r)
x, frames = test_solver(rho_l,rho_r,v_l,v_r,riemann.traffic_vc_1D)

def plot_frame(t):
    ax = riemann_tools.plot_riemann(states, speeds, reval,
                       wave_types, t, layout='horizontal',
                       extra_axes=True);
    riemann_tools.plot_characteristics(reval,c,(v_l,v_r),ax[0],
                                       extra_lines=[[[0,0],[0,1]]])
    rho = frames[int(t*10)].q[0,:]
    ax[1].plot(x,rho,'-sg')
    ax[1].set_xlim(-0.7, 0.7)
    ax[1].legend(['Exact','Approx'],loc='upper right',fontsize=10);
    traffic_variable_speed.phase_plane_plot(rho_l,rho_r,
                                            v_l,v_r,axes=ax[2])
    plt.show()

interact(plot_frame,
         t=widgets.FloatSlider(min=0,max=1,step=0.1,value=0.5));
```

### $f$-wave solvers
The use of the full jump $\rho_r-\rho_l$ in the wave discussed above may seem strange since part of this jump remains stationary at the interface (and part of it may be sent in the opposite direction!)  For systems where the flux depends explicitly on $x$, it is often simplest to use the $f$-wave approach; for a scalar problem this corresponds to using the jump in the flux, $f_r-f_l$ in place of the product $s(\rho_r-\rho_l)$.

This technique can be used in the present system, but it has a small drawback.  At the center of a transonic rarefaction, the cell average is $1/2$ and the $f$-wave $f_r-f_l$ will be essentially zero.  This causes the limiter to activate and reduce the method locally to first order, with the consequence that the solution is inaccurate near the sonic point.  The first approach discussed above (where the wave strength is $\rho_r-\rho_l$) is free from this deficiency.

```python
rho_l = 0.7; rho_r = 0.2
v_l = 1.0; v_r = 1.0
t = 0.5

states, speeds, reval, wave_types = \
    traffic_variable_speed.exact_riemann_solution(rho_l,rho_r,
                                                  v_l,v_r)
x, frames  = test_solver(rho_l,rho_r,v_l,v_r,
                         riemann.traffic_vc_1D)
x ,fframes = test_solver(rho_l,rho_r,v_l,v_r,
                         riemann.traffic_vc_fwave_1D)

def plot_frame(t):
    rho = frames[int(t*10)].q[0,:]
    rho2 = fframes[int(t*10)].q[0,:]
    ax = riemann_tools.plot_riemann(states, speeds,
                       reval, wave_types, t, layout='horizontal',
                       extra_axes=True);
    riemann_tools.plot_characteristics(reval,c,(v_l,v_r),ax[0])
    ax[1].plot(x,rho,'-sg')
    ax[1].plot(x,rho2,'-or')
    ax[1].legend(['Exact','Approx','$f$-wave'],loc='upper right',
                 fontsize=10);
    ax[1].set_xlim(-0.2,0.2)
    traffic_variable_speed.phase_plane_plot(rho_l,rho_r,v_l,v_r,
                                            axes=ax[2])
    plt.show()

interact(plot_frame,
         t=widgets.FloatSlider(min=0,max=1,step=0.1,value=0.5));
```


# NODDI-Watson

*(Zhang et al. 2012)* proposed a model to estimate the dispersion of neurites (i.e. both axons and neuron dendrites), and is called Neurite Orientation Dispersion and Density Imaging (NODDI). It models dispersion for a single axon bundle using a Watson distribution $W(\kappa,\boldsymbol{\mu})$, that is a particular case of Bingham when $\kappa=\kappa_1=\kappa_2$. As such, it is incapable of modeling axon bending or fanning. The model separately models the CSF, restricted and hindered diffusion as a Gaussian Ball, Stick and Zeppelin model. NODDI models the signal as
\begin{align}
 E^{\textrm{NODDI}}_{\textrm{Watson}}= \underbrace{f_{\textrm{CSF}}\overbrace{E_{\textrm{iso}}(\cdot|\lambda_{\textrm{CSF}})}^{\textrm{Ball}}}_{\textrm{CSF}}
                             +\overbrace{W(\kappa,\boldsymbol{\mu})}^{\textrm{Watson}}\,*_{\mathbb{S}^2}\, \left[\underbrace{f_h\overbrace{E_{\textrm{h}}(\cdot|\lambda_\perp^{\textrm{tort}},\lambda_\parallel)}^{\textrm{Zeppelin}}}_{\textrm{Hindered Extra-Axonal}}+\underbrace{f_r\overbrace{E_r(\cdot|\lambda_\parallel)}^{\textrm{Stick}}}_{\textrm{Intra-Axonal}}\right].
\end{align}
To improve the stability of estimated model parameters, NODDI sets several constraints on model parameters:
- isotropic diffusivity $\lambda_{\textrm{CSF}}$ is fixed to $3\cdot10^-9 m^2/s$
- parallel diffusivity $\lambda_\parallel$ of the stick and zeppelin are the same and fixed to $1.7\cdot10^-9 m^2/s$
- A tortuosity model is used to link the perpendicular diffusivity $\lambda_\perp^{\textrm{tort}}$ to the intra- and extra- axonal volume fraction $f_r$, $f_h$ and parallel diffusivity $\lambda_\parallel$.

The only parameters that NODDI-Watson finally fits to the data are Watson parameters $\kappa$, $\mu$ and the volume fractions $f_{\textrm{CSF}}$, $f_h$ and $f_r$.

# Using Dmipy to set up the NODDI-Watson model

Setting up the NODDI-Watson model in Dmipy is straight-forward.
First, we instantiate the model components for the non-dispersed components themselves: a Ball for the CSF, and the Stick and Zeppelin for the intra- and extra-axonal diffusion.

```python
from dmipy.signal_models import cylinder_models, gaussian_models
ball = gaussian_models.G1Ball()
stick = cylinder_models.C1Stick()
zeppelin = gaussian_models.G2Zeppelin()
```

We then Watson-disperse the stick and zeppelin together, making a representation for a dispersed single axon bundle

```python
from dmipy.distributions.distribute_models import SD1WatsonDistributed
watson_dispersed_bundle = SD1WatsonDistributed(models=[stick, zeppelin])
```

Then, we can set part of NODDI's model parameter links that we listed above. The bundle representation allows to set tortuous, equal and fixed parameters. First, we show the parameter names of the bundle.

```python
watson_dispersed_bundle.parameter_names
```

    ['G2Zeppelin_1_lambda_perp',
     'SD1Watson_1_odi',
     'G2Zeppelin_1_lambda_par',
     'SD1Watson_1_mu',
     'C1Stick_1_lambda_par',
     'partial_volume_0']

We can then set the tortuous $\lambda_\perp$ parameter, followed by the equal $\lambda_\parallel$ parameters of the stick and zeppelin, and finally we set the remaining $\lambda_\parallel$ to $1.7\times10^{-9}m^2/s$. Note that fixing or linking a parameter removes it permanently from the parameter list, meaning it cannot be used for subsequent parameter links. For this reason, we needed to do the tortuosity constraint before fixing $\lambda_\parallel$.

```python
watson_dispersed_bundle.set_tortuous_parameter('G2Zeppelin_1_lambda_perp','C1Stick_1_lambda_par','partial_volume_0')
watson_dispersed_bundle.set_equal_parameter('G2Zeppelin_1_lambda_par', 'C1Stick_1_lambda_par')
watson_dispersed_bundle.set_fixed_parameter('G2Zeppelin_1_lambda_par', 1.7e-9)
```

Finally, we put the model together with the scheme as follows.

```python
from dmipy.core.modeling_framework import MultiCompartmentModel
NODDI_mod = MultiCompartmentModel(models=[ball, watson_dispersed_bundle])
```

You can see the names and cardinality of the to-be optimized parameters in the generated MultiCompartmentModel as follows.

```python
NODDI_mod.parameter_names
```

    ['G1Ball_1_lambda_iso',
     'SD1WatsonDistributed_1_SD1Watson_1_mu',
     'SD1WatsonDistributed_1_partial_volume_0',
     'SD1WatsonDistributed_1_SD1Watson_1_odi',
     'partial_volume_0',
     'partial_volume_1']

The last thing is to fix the diffusivity of the Ball compartment to static values. We can do that using the model.set_fixed_parameter(parameter_name, value) function.

```python
NODDI_mod.set_fixed_parameter('G1Ball_1_lambda_iso', 3e-9)
```

We can visualize the model as follows:

```python
from IPython.display import Image
NODDI_mod.visualize_model_setup(view=False, cleanup=False)
Image('Model Setup.png')
```

# Fitting NODDI to Human Connectome Project data

To illustrate Dmipy's ease-of-use we also fit the our constructed model to a coronal slice of the HCP data. We will illustrate FODs in the red region of interest later.

```python
from dmipy.data import saved_data
scheme_hcp, data_hcp = saved_data.wu_minn_hcp_coronal_slice()
```

    This data slice originates from Subject 100307 of the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.

```python
import matplotlib.pyplot as plt
import matplotlib.patches as patches
%matplotlib inline

fig, ax = plt.subplots(1)
ax.imshow(data_hcp[:, 0, :, 0].T, origin=True)
rect = patches.Rectangle((70,70),20,20,linewidth=1,edgecolor='r',facecolor='none')
ax.add_patch(rect)
ax.set_axis_off()
ax.set_title('HCP coronal slice B0 with ROI');
```

## Fit NODDI to HCP data
Having pathos and numba installed, fitting the model to this coronal slice takes less than 10 minutes.

```python
NODDI_fit_hcp = NODDI_mod.fit(
    scheme_hcp, data_hcp, mask=data_hcp[..., 0]>0)
```

    Using parallel processing with 8 workers.
    Setup brute2fine optimizer in 58.6642770767 seconds
    Fitting of 8181 voxels complete in 1131.65072489 seconds.
    Average of 0.138326699045 seconds per voxel.

## Visualize results

```python
fitted_parameters = NODDI_fit_hcp.fitted_parameters

fig, axs = plt.subplots(2, 2, figsize=[15, 15])
axs = axs.ravel()

counter = 0
for name, values in fitted_parameters.items():
    if values.squeeze().ndim != 2:
        continue
    cf = axs[counter].imshow(values.squeeze().T, origin=True, interpolation='nearest')
    axs[counter].set_axis_off()
    axs[counter].set_title(name)
    fig.colorbar(cf, ax=axs[counter], shrink=0.7)
    counter += 1
```

Above we see the familiar maps. Remember that partial_volume_0/1 represent fractions of models in the order they were given to the MultiCompartmentMicrostructureModel object: in this case, 0 represents the fraction of Ball (CSF), and the 1 represents the Watson Bundle (white matter). The `SD1WatsonDistributed_1_partial_volume_0` is the normalized volume fraction of the Stick **within the WatsonBundle**.

To get the signal contribution of the Stick (intra-axonal) to the total signal, we must multiply `partial_volume_1 * SD1WatsonDistributed_1_partial_volume_0`. To get the signal contribution of the zeppelin (extra-axonal), we must multiply `partial_volume_1 * (1-SD1WatsonDistributed_1_partial_volume_0)`.

In the bottom-right ODI map you can see that the concentration is highest in the corpus callosum and in coherent white matter bundles. Finally, the $\lambda_\parallel$ and $\lambda_{\textrm{iso}}$ maps are fixed to the values we gave them.

```python
# get total Stick signal contribution
vf_intra = (fitted_parameters['SD1WatsonDistributed_1_partial_volume_0'] *
            fitted_parameters['partial_volume_1'])
# get total Zeppelin signal contribution
vf_extra = ((1 - fitted_parameters['SD1WatsonDistributed_1_partial_volume_0']) *
            fitted_parameters['partial_volume_1'])
fig, axs = plt.subplots(1, 2, figsize=[15, 7])
axs = axs.ravel()

counter = 0
for name, values in zip(['Total vf_intra - Stick', 'Total vf_extra - Zeppelin'], [vf_intra, vf_extra]):
    cf = axs[counter].imshow(values.squeeze().T, origin=True, interpolation='nearest')
    axs[counter].set_axis_off()
    axs[counter].set_title(name)
    fig.colorbar(cf, ax=axs[counter], shrink=0.7)
    counter += 1
```

## Estimating Fiber Orientation Distributions (FODs)

We estimate FODs on the right side of the corpus callosum and use the estimated concentration ODI as background.

```python
from dipy.data import get_sphere
from dipy.viz import window, actor
sphere = get_sphere(name='symmetric724').subdivide()
# The next line produces the FOD sphere functions.
# The lower bound option artificially puts a lower bound on the ODI so they don't get too sharp.
# It does not affect the fitted_parameters themselves.
fods = NODDI_fit_hcp.fod(sphere.vertices, visual_odi_lower_bound=0.08)[70:90,: , 70:90]
```

```python
import numpy as np
affine = np.eye(4)
volume_res = fitted_parameters['SD1WatsonDistributed_1_SD1Watson_1_odi'][70:90,: , 70:90]
volume_im = actor.slicer(volume_res[:, 0, :, None], interpolation='nearest', affine=affine, opacity=0.7)
```

```python
ren = window.Renderer()
fod_spheres = actor.odf_slicer(
    fods, sphere=sphere, scale=0.9, norm=False)
fod_spheres.display_extent(0, fods.shape[0]-1, 0, fods.shape[1]-1, 0, fods.shape[2]-1)
fod_spheres.RotateX(90)
fod_spheres.RotateZ(180)
fod_spheres.RotateY(180)
ren.add(fod_spheres)
ren.add(volume_im)
window.record(ren, size=[700, 700])
```

```python
import matplotlib.image as mpimg
img = mpimg.imread('fury.png')

plt.figure(figsize=[10, 10])
plt.imshow(img[100:-97, 100:-85])
plt.title('NODDI-Watson FODs with ODI background', fontsize=20)
plt.axis('off');
```

You can see the corpus callosum coming in the from the left and joining the centrum semiovale on the right. The intensity of the background shows ODI of the Watson distribution (inversely related to concentration $\kappa$). Notice it is low in the corpus callosum and higher in non-white matter areas. Notice that NODDI-Watson cannot estimate crossings, which are expected in the centrum semiovale on the right. This biases any comparison/estimation of volume fractions and concentrations in these areas.

```python
from os import remove
remove('fury.png')
```

## Estimating Error Metrics: MSE and $R^2$

Whenever you fit a model, it's always wise to check out the error of the model fit. Here we estimate the Mean Squared Error (MSE) and the $R^2$ coefficient of determination. In MSE the lower is better, while $R^2$ has an upper bound of one, which represents a perfect model fit.

```python
mse = NODDI_fit_hcp.mean_squared_error(data_hcp)
R2 = NODDI_fit_hcp.R2_coefficient_of_determination(data_hcp)

fig, axs = plt.subplots(1, 2, figsize=[15, 15])
cf = axs[0].imshow(mse.squeeze().T, origin=True, vmax=0.02, cmap='Greys_r')
axs[0].set_axis_off()
fig.colorbar(cf, ax=axs[0], shrink=0.33)
axs[0].set_title('Mean Squared Error', fontsize=20)
cf = axs[1].imshow(R2.squeeze().T, origin=True, vmin=0.6, cmap='Greys_r')
axs[1].set_axis_off()
fig.colorbar(cf, ax=axs[1], shrink=0.33)
axs[1].set_title('R2 - Coefficient of Determination', fontsize=20);
```

Ideally, you want the MSE and $R^2$ to show no structure of the underlying data - which would mean the model can at least fit the data equally well everywhere. Instead, in NODDI-Watson we can see clearly see the structure of the different brain areas. This means that the model cannot fit the diffusion signal equally well in all brainn areas, and that interpretation of model parameters will be biased when comparing them between these different areas.

When doing these studies, ALWAYS verify whether differences in estimated parameters reflect actual tissue changes, or only reflect the limitations of the model itself.

## References
- Zhang, Hui, et al. "NODDI: practical in vivo neurite orientation dispersion and density imaging of the human brain." Neuroimage 61.4 (2012): 1000-1016.


# Chi-Squared Distribution
***
## Definition
>The Chi-Squared distribution is a continous probability distribution focused on sample standard deviations and can (e.g.) "let you know whether two groups have significantly different opinions, which makes it a very useful statistic for survey research"  $ ^{[1]}$.

## Formula
The probability mass function of a Chi-Squared distributed random variable is defined as:
$$
 \begin{equation}
    f(x|k) =
    \begin{cases}
      \frac{x^{\frac{k}{2}-1}e^{-\frac{x}{2}}}{2^{\frac{k}{2}}\Gamma(\frac{k}{2})}, & \text{if}\ x>0 \\
      0, & \text{otherwise}
    \end{cases}
  \end{equation}
$$

where
$$ \Gamma(n) = (n-1)! $$

and $k$ denotes the degrees of freedom.

```python
# IMPORTS
import numpy as np
import scipy.stats as stats
import matplotlib.pyplot as plt
import matplotlib.style as style
from IPython.core.display import HTML

# PLOTTING CONFIG
%matplotlib inline
style.use('fivethirtyeight')
plt.rcParams["figure.figsize"] = (14, 7)
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: center;
}
</style>
""")
plt.figure(dpi=100)

# PDF
plt.plot(np.linspace(0, 20, 100),
         stats.chi2.pdf(np.linspace(0, 20, 100), df=4) / np.max(stats.chi2.pdf(np.linspace(0, 20, 100), df=4)),
        )
plt.fill_between(np.linspace(0, 20, 100),
                 stats.chi2.pdf(np.linspace(0, 20, 100), df=4) / np.max(stats.chi2.pdf(np.linspace(0, 20, 100), df=4)),
                 alpha=.15,
                )

# CDF
plt.plot(np.linspace(0, 20, 100),
         stats.chi2.cdf(np.linspace(0, 20, 100), df=4),
        )

# LEGEND
plt.xticks(np.arange(0, 21, 2))
plt.text(x=11, y=.25, s="pdf (normed)", alpha=.75, weight="bold", color="#008fd5")
plt.text(x=11, y=.85, s="cdf", alpha=.75, weight="bold", color="#fc4f30")

# TICKS
plt.xticks(np.arange(0, 21, 2))
plt.tick_params(axis = 'both', which = 'major', labelsize = 18)
plt.axhline(y = 0, color = 'black', linewidth = 1.3, alpha = .7)

# TITLE, SUBTITLE & FOOTER
plt.text(x = -2, y = 1.25, s = r"Chi-Squared $(\chi^{2})$ Distribution - Overview",
               fontsize = 26, weight = 'bold', alpha = .75)
plt.text(x = -2, y = 1.1,
         s = 'Depicted below are the normed probability density function (pdf) and the cumulative density\nfunction (cdf) of a Chi-Squared distributed random variable $ y \sim \chi^{2}(k) $, given $k$=4.',
         fontsize = 19, alpha = .85)
plt.text(x = -2,y = -0.2,
         s = 'Chi-Square',
         fontsize = 14, color = '#f0f0f0', backgroundcolor = 'grey');
```

***
## Parameters

```python
# IMPORTS
import numpy as np
import scipy.stats as stats
import matplotlib.pyplot as plt
import matplotlib.style as style
from IPython.core.display import HTML

# PLOTTING CONFIG
%matplotlib inline
style.use('fivethirtyeight')
plt.rcParams["figure.figsize"] = (14, 7)
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: center;
}
</style>
""")
plt.figure(dpi=100)

# PDF k = 1
plt.plot(np.linspace(0, 15, 500),
         stats.chi2.pdf(np.linspace(0, 15, 500), df=1),
        )
plt.fill_between(np.linspace(0, 15, 500),
                 stats.chi2.pdf(np.linspace(0, 15, 500), df=1),
                 alpha=.15,
                )

# PDF k = 3
plt.plot(np.linspace(0, 15, 100),
         stats.chi2.pdf(np.linspace(0, 15, 100), df=3),
        )
plt.fill_between(np.linspace(0, 15, 100),
                 stats.chi2.pdf(np.linspace(0, 15, 100), df=3),
                 alpha=.15,
                )

# PDF k = 6
plt.plot(np.linspace(0, 15, 100),
         stats.chi2.pdf(np.linspace(0, 15, 100), df=6),
        )
plt.fill_between(np.linspace(0, 15, 100),
                 stats.chi2.pdf(np.linspace(0, 15, 100), df=6),
                 alpha=.15,
                )

# LEGEND
plt.text(x=.5, y=.7, s="$ k = 1$", rotation=-65, alpha=.75, weight="bold", color="#008fd5")
plt.text(x=1.5, y=.35, s="$ k = 3$", alpha=.75, weight="bold", color="#fc4f30")
plt.text(x=5, y=.2, s="$ k = 6$", alpha=.75, weight="bold", color="#e5ae38")

# TICKS
plt.tick_params(axis = 'both', which = 'major', labelsize = 18)
plt.axhline(y = 0, color = 'black', linewidth = 1.3, alpha = .7)

# TITLE, SUBTITLE & FOOTER
plt.text(x = -1.5, y = 2.8, s = "Chi-Squared Distribution - $ k $",
               fontsize = 26, weight = 'bold', alpha = .75)
plt.text(x = -1.5, y = 2.5,
         s = 'Depicted below are three Chi-Squared distributed random variables with varying $ k $. As one can\nsee the parameter $k$ smoothens the distribution and softens the skewness.',
         fontsize = 19, alpha = .85)
plt.text(x = -1.5,y = -0.4,
         s = 'Chi-Square',
         fontsize = 14, color = '#f0f0f0', backgroundcolor = 'grey');
```

***
## Implementation in Python
Multiple Python packages implement the Chi-Squared distribution. One of those is the stats.poisson module from the scipy package. The following methods are only an excerpt. For a full list of features the [official documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.poisson.html) should be read.

### Random Variates
In order to generate a random sample from, the function `rvs` should be used.

```python
import numpy as np
from scipy.stats import chi2

# draw a single sample
np.random.seed(42)
print(chi2.rvs(df=4), end="\n\n")

# draw 10 samples
print(chi2.rvs(df=4, size=10), end="\n\n")
```

    4.78735877974

    [ 2.98892946  2.76456717  2.76460459  9.29942882  5.73341246  2.262156
      4.93962895  3.99792053  0.43182989  1.34248457]

### Probability Density Function
The probability mass function can be accessed via the `pdf` function. Like the rvs method, the pdf allows for adjusting the mean of the random variable:

```python
from scipy.stats import chi2

# additional imports for plotting purpose
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams["figure.figsize"] = (14,7)

# likelihood of x and y
x = 1
y = 7
print("pdf(1) = {}\npdf(7) = {}".format(chi2.pdf(x=x, df=4), chi2.pdf(x=y, df=4)))

# continuous pdf for the plot
x_s = np.arange(15)
y_s = chi2.pdf(x=x_s, df=4)
plt.scatter(x_s, y_s, s=100);
```

### Cumulative Probability Density Function¶
The cumulative probability density function is useful when a probability range has to be calculated. It can be accessed via the `cdf` function:

```python
from scipy.stats import chi2

# probability of x less or equal 0.3
print("P(X <=3) = {}".format(chi2.cdf(x=3, df=4)))

# probability of x in [-0.2, +0.2]
print("P(2 < X <= 8) = {}".format(chi2.cdf(x=8, df=4) - chi2.cdf(x=2, df=4)))
```

    P(X <=3) = 0.4421745996289252
    P(2 < X <= 8) = 0.6441806878992138

***
## Infering $k$
Given a sample of datapoints it is often required to estimate the "true" parameters of the distribution. In the case of the Chi-Squared distribution this estimation is quite simple. $k$ can be derived by calculating the mean of the sample.

```python
# IMPORTS
import numpy as np
import scipy.stats as stats
import matplotlib.pyplot as plt
import matplotlib.style as style
from IPython.core.display import HTML

# PLOTTING CONFIG
%matplotlib inline
style.use('fivethirtyeight')
plt.rcParams["figure.figsize"] = (14, 7)
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: center;
}
</style>
""")
plt.figure(dpi=100)

##### COMPUTATION #####
# DECLARING THE "TRUE" PARAMETERS UNDERLYING THE SAMPLE
k_real = 2

# DRAW A SAMPLE OF N=1000
np.random.seed(42)
sample = stats.chi2.rvs(df=k_real, size=1000)

# ESTIMATE K
k_est = np.mean(sample)
print("Estimated k: {}".format(k_est))

##### PLOTTING #####
# SAMPLE DISTRIBUTION
plt.hist(sample, bins=50, alpha=.25)

# TRUE CURVE
plt.plot(np.linspace(0, 18, 1000), stats.chi2.pdf(np.linspace(0, 18, 1000),df=k_real))

# ESTIMATED CURVE
plt.plot(np.linspace(0, 18, 1000), stats.chi2.pdf(np.linspace(0, 18, 1000),df=k_est))

# LEGEND
plt.text(x=.75, y=.1, s="sample", alpha=.75, weight="bold", color="#008fd5")
plt.text(x=3, y=.15, s="true distrubtion", alpha=.75, weight="bold", color="#fc4f30")
plt.text(x=1, y=.4, s="estimated distribution", alpha=.75, weight="bold", color="#e5ae38")

# TICKS
plt.xticks(range(0, 19)[::2])
plt.tick_params(axis = 'both', which = 'major', labelsize = 18)
plt.axhline(y = 0.003, color = 'black', linewidth = 1.3, alpha = .7)

# TITLE, SUBTITLE & FOOTER
plt.text(x = -2, y = 0.675, s = "Chi-Squared Distribution - Parameter Estimation",
               fontsize = 26, weight = 'bold', alpha = .75)
plt.text(x = -2, y = 0.6,
         s = 'Depicted below is the distribution of a sample (blue) drawn from a Chi-Squared distribution with \n$k=2$ (red). Also the estimated distrubution with $k \sim {:.3f} $ is shown (yellow).'.format(np.mean(sample)),
         fontsize = 19, alpha = .85)
plt.text(x = -2,y = -0.075,
         s = 'Chi-Square',
         fontsize = 14, color = '#f0f0f0', backgroundcolor = 'grey');
```

## Infering $k$ - MCMC
In addition to a "direct" inference, $k$ can also be estimated using Markov chain Monte Carlo simulation - implemented in Python's [PyMC3](https://github.com/pymc-devs/pymc3).

```python
# IMPORTS
import numpy as np
import pymc3 as pm
import scipy.stats as stats
import matplotlib.pyplot as plt
import matplotlib.style as style
from IPython.core.display import HTML

# PLOTTING CONFIG
%matplotlib inline
style.use('fivethirtyeight')
plt.rcParams["figure.figsize"] = (14, 7)
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: center;
}
</style>
""")
plt.figure(dpi=100)

##### COMPUTATION #####
# DECLARING THE "TRUE" PARAMETERS UNDERLYING THE SAMPLE
k_real = 2

# DRAW A SAMPLE OF N=1000
np.random.seed(42)
sample = stats.chi2.rvs(df=k_real, size=1000)

##### SIMULATION #####
# MODEL BUILDING
with pm.Model() as model:
    k = pm.DiscreteUniform("k", lower=0, upper=np.mean(sample)*7) # mean + 3stds
    chi_2 = pm.ChiSquared("chi2", nu=k, observed=sample)

# MODEL RUN
with model:
    trace = pm.sample(50000)
    burned_trace = trace[45000:]

# MU - 95% CONF INTERVAL
ks = burned_trace["k"]
k_est_95 = np.mean(ks) - 2*np.std(ks), np.mean(ks) + 2*np.std(ks)
print("95% of sampled mus are between {} and {}".format(*k_est_95))

##### PLOTTING #####
# SAMPLE DISTRIBUTION
plt.hist(sample, bins=50,normed=True, alpha=.25)

# TRUE CURVE
plt.plot(np.linspace(0, 18, 1000), stats.chi2.pdf(np.linspace(0, 18, 1000),df=k_real), linestyle="--")

# ESTIMATED CURVE
plt.plot(np.linspace(0, 18, 1000), stats.chi2.pdf(np.linspace(0, 18, 1000),df=np.mean(ks)), linestyle=":")

# LEGEND
plt.text(x=.75, y=.1, s="sample", alpha=.75, weight="bold", color="#008fd5")
plt.text(x=3, y=.15, s="true distrubtion", alpha=.75, weight="bold", color="#fc4f30")
plt.text(x=1, y=.4, s="estimated distribution", alpha=.75, weight="bold", color="#e5ae38")

# TICKS
plt.xticks(range(0, 19)[::2])
plt.tick_params(axis = 'both', which = 'major', labelsize = 18)
plt.axhline(y = 0.003, color = 'black', linewidth = 1.3, alpha = .7)

# TITLE, SUBTITLE & FOOTER
plt.text(x = -2, y = 0.675, s = "Chi-Squared Distribution - Parameter Estimation (MCMC)",
               fontsize = 26, weight = 'bold', alpha = .75)
plt.text(x = -2, y = 0.6,
         s = 'Depicted below is the distribution of a sample (blue) drawn from a Chi-Squared distribution with \n$k=2$ (red). Also the estimated distrubution with $k \sim {} $ is shown (yellow).'.format(np.mean(ks)),
         fontsize = 19, alpha = .85)
plt.text(x = -2,y = -0.075,
         s = 'Chi-Square',
         fontsize = 14, color = '#f0f0f0', backgroundcolor = 'grey');
```

***
[1] - [Practical Surveys. Understanding Chi Squared](http://practicalsurveys.com/reporting/chisquare.php)

Mathematics/Statistics/Statistics and Probability Python Notebooks/Important-Statistics-Distributions-py-notebooks/Chi-Squared Distribution.ipynb