# Introduction

Many of the models that we have considered up to this point involved two components: (1) a set of perceptual-motor processes and (2) a single memory retrieval. To simplify the model, the perceptual-motor processing time was treated as a constant rather than a random variable. Although this type of model may work in many situations, there are situations in which more complex models need to be developed. How do we develop a model in cases where there are multiple memory retrievals associated with a single response or in cases where we want to treat the perceptual-motor processes as a random variable instead of a constant? In this tutorial, we will use a method called fast Fourier transform to convolve or sum multiple random variables. We will illustrate how to use fast Fourier transform for convolution using the Count Model from the Lisp ACT-R tutorial, which entails a series of memory retrievals that culminate into an observed response.

## Count Task

At the begining of a trial, two numbers are displayed on the screen: (1) a start number, and (2) an end number. Th task requires a person to count from the start number to the end number and press a key upon completion. We will designate 2 as the start number and 4 as the end number.

# Count Model

## Overview

The count model begins with a set of production rules that encode the start and end number displayed on a screen. The first production rule sets the goal to count, finds the stimulus, and creates a chunk in the imaginal buffer to hold the start and end numbers. The next production rule shifts visual attention the stimulus for encoding. Next, a production rule retrieves the meaning of the number. Note that because base-level activation is set to 10, retrieval times are neglible. Finally, the value associated with the value slot of the retrieved chunk is transferred to the imaginal buffer where it is added to the start slot. A similar process is repeated for the end number stimulus.

Once the stimuli are encoded into the chunk in the imaginal buffer, the model proceeds through a series of retrievals of "count-order" chunks to increment from the start number to the end number. Each "count-order" chunk contains a slot for the current number and the next number. The counting process begins with the selection of a production rule that issues a retrieval request for a "count-order" chunk with a current slot value equal to the start number. Upon retrieving the "count-order" chunk, an increment count production rule is fired to increment the count. This production sets the current count to the the value of the "next" slot of the retrieved chunk and issues new retrieval request using the same "next" value for the "current" slot. In essence, this creates a loop or chain in which the count is incremented and the subsequent chunk is retrieved until the end number is reached. When the end number is reached, a "stop" production rule issues a command to press the "j" key.

## Declarative Memory

Declarative memory $M$ consists of chunks with slots $Q = \rm \{current, next\}$. The slot "current" holds the value of the current number in the counting process. The slot "next" holds the number that immediately follows the current number. For example, chunk $m$ could be defined as:

\begin{align}
\mathbf{c}_m = \rm \{(current,1),(next,2) \}
\end{align}

## Activation

Activation for chunk $m$ is defined as:

$a_m = \rm blc + \epsilon_m$

where blc is the base level constant and $\epsilon_m \sim \rm Normal(0,\sigma)$. Note that $E[a_m] = \mu_m$. Given that numbers are well-learned, we assume that confusions are not possible and that the correct number is always retrieved.

## Retrieval Request

When incrementing the count, a new retrieval request is defined as

\begin{align}
\mathbf{r} = \rm \{current, c_{r,\textrm{retrieval}}(next)\}
\end{align}

where $\mathbf{c}_{r,\textrm{retrieval}}$ is the chunk in the retrieval buffer. The result of the retrieval request becomes the new chunk in the retrieval buffer, $\mathbf{c}_{j,\textrm{retrieval}}$. For example, if $\rm \mathbf{c}_{r,\textrm{retrieval}} = \{\rm(current,1),(next,2) \}$, then $\mathbf{r} = \{\rm(current,2)\}$ and the new chunk $\rm \mathbf{c}_{j,\textrm{retrieval}} = \{\rm(current,2),(next,3)\}$.

## Response Mapping

The model will respond once the current value of the chunk in the retrieval buffer is equal to the goal number of 4. Formally, we define the response mapping as

\begin{equation}
 \rm p_i(end) = c_{r,retrieval}(current)
\end{equation}

where $\mathbf{p_i} = \rm \{(end,4)\}$ is production rule $i$ and the slot "end" holds the goal state of 4.

## Convolution

Our basic approach will be to use a normal approximation to convolve conflict resolution and perceptual-motor time and to use FFT to convolve memory retrieval times with the sum of the conflict resolution and perceptual-motor time.

In order to provide some intuition behind convolution, it might be helpful to consider the trace from the computational version of the count model:

     Time    Module                 Status
     0.000   GOAL                   SET-BUFFER-CHUNK GOAL CURRENT-GOAL REQUESTED NIL
     0.000   VISION                 SET-BUFFER-CHUNK VISUAL-LOCATION VISUAL-LOCATION0-0 REQUESTED NIL
     0.042   PROCEDURAL             PRODUCTION-FIRED START
     0.042   VISION                 SET-BUFFER-CHUNK VISUAL-LOCATION VISUAL-LOCATION0-0
     0.179   IMAGINAL               SET-BUFFER-CHUNK IMAGINAL QUESTION0
     0.244   PROCEDURAL             PRODUCTION-FIRED ATTEND-VISUAL-LOCATION
     0.333   VISION                 SET-BUFFER-CHUNK VISUAL TEXT0
     0.399   PROCEDURAL             PRODUCTION-FIRED RETRIEVE-MEANING
     0.399   DECLARATIVE            SET-BUFFER-CHUNK RETRIEVAL TWO
     0.443   PROCEDURAL             PRODUCTION-FIRED ENCODE-START-NUMBER
     0.443   VISION                 SET-BUFFER-CHUNK VISUAL-LOCATION VISUAL-LOCATION1-0
     0.499   PROCEDURAL             PRODUCTION-FIRED ATTEND-VISUAL-LOCATION
     0.596   VISION                 SET-BUFFER-CHUNK VISUAL TEXT1
     0.641   PROCEDURAL             PRODUCTION-FIRED RETRIEVE-MEANING
     0.641   DECLARATIVE            SET-BUFFER-CHUNK RETRIEVAL FOUR
     0.698   PROCEDURAL             PRODUCTION-FIRED ENCODE-END-NUMBER
     0.757   PROCEDURAL             PRODUCTION-FIRED START-COUNT
     1.012   DECLARATIVE            SET-BUFFER-CHUNK RETRIEVAL C
     1.070   PROCEDURAL             PRODUCTION-FIRED INCREMENT
     1.381   DECLARATIVE            SET-BUFFER-CHUNK RETRIEVAL D
     1.431   PROCEDURAL             PRODUCTION-FIRED INCREMENT
     1.490   PROCEDURAL             PRODUCTION-FIRED STOP
     1.490   MOTOR                  PRESS-KEY j

As you can see, current model time increments each time a new cognitive process is executed. The processing time for each of these processes is summed to produce a reaction time of 1.49 seconds. In summary, we sum 11 conflict resolutions, 2 visual attention shifts, 1 chunk creation in the imaginal buffer, 2 memory retrievals, and 1 motor execution.

## Conflict resolution and perceptual-motor time

In ACT-R, conflict resolution and perceptual motor processing time (i.e. visual attention, motor execution) are often fixed to default values as a simplifying assumption. However, it is possible to treat conflict resolution and perceptual motor time as a random variable. ACT-R takes a somewhat unconventional approach of using uniform distributions each distribution. A Uniform distribution is defined by two parameters:

- $u_{\rm min}$: the minumum value
- $u_{\rm max}$: the maximum value

In ACT-R, parameters $u_{\rm min}$ and $u_{\rm max}$ are defined in terms of the mean processing time $\mu$ as follows:

\begin{align}
u_\text{min} = \mu - \frac{\mu}{3}
\end{align}

\begin{align}
u_\text{max} = \mu + \frac{\mu}{3}
\end{align}

These definitions indicate the range and variance of processing time increases as a function of mean processing time.

### Normal Approximation

The sum of Uniform random variables is a [Irwin-Hall distribution
](https://en.wikipedia.org/wiki/Irwin%E2%80%93Hall_distribution). Unfortunately, the Irwin-Hall has at least two limitations. First, computing the pdf is more computationally expensive compared to the pdf of alternative distributions, even in the simple case where $u_{\rm min} = 0$ and $u_{\rm max} = 1$ . Second, the pdf becomes more complex in the general case in which the parameters differ.

Fortunately, the Irwin-Hall distribution can be easily approximated with a normal distribution as long as the standard deviations of the component distributions are not drastically different. The code block below provides an illustration of the approximation. It is clear that the approximation works well with as few as three random variables.

```julia
cd(@__DIR__)
using Pkg
Pkg.activate("../../")
using StatsPlots, Random, ACTRModels, CSV, FFTDists, DifferentialEvolutionMCMC
using MCMCChains
include("Count.jl")
Random.seed!(76121);
```

      Activating environment at `~/.julia/dev/ACTRFundamentalTools/Project.toml`

```julia
# simulate sum of n uniform random variables representing conflict resolution time
sum_n(n) = sum(rand(Uniform(0.033,0.0667), 10^5, n), dims=2)

# normal approximation of n uniform random variables
function normal_density(n)
    μ,σ = convolve_normal(;cr=(μ=0.05,N=n))
    x = range(-3*σ + μ, 3*σ + μ, length = 100)
    d = pdf.(Normal(μ,σ), x)
    return x,d
end

# simulate sum of n uniform random variables
samples = map(n -> sum_n(n), 1:4)
# density of normal approximation
density = map(n -> normal_density(n), 1:4)
pyplot()
#  plot histograms
hist_plot = histogram(samples, grid=false, layout=(2,2), title = ["1" "2" "3" "4"], leg=false, color=:darkgrey,
    norm=true, size=(800,600))
# superimpose densities
plot!(hist_plot, density, linewidth=1.5)
```

In order to use the Normal approximation, we must first define the mean and standard deviation of a Uniform distribution. The mean of a Uniform distribution is:

\begin{align}
\mu = \frac{u_\text{max}-u_\text{min}}{2}
\end{align}

The standard deviation of a Uniform distribution is:

\begin{align}
\sigma = \sqrt{\frac{1}{12}\left(u_\text{max}-u_{min}\right)^2}
\end{align}

After subtituting the definitions of $u_{\rm min}$ and $u_{\rm max}$ and simplifying, we get the following for the standard deviation for ACT-R:

\begin{align}
\sigma = \sqrt{\frac{1}{12}\left(\mu + \frac{\mu}{3} - \left(\mu - \frac{\mu}{3}\right) \right)^2} = \frac{\mu}{3}\sqrt{\frac{1}{3}}
\end{align}

We will fix mean processing times for conflict resolution, attention latency, imaginal latency, and motor execution to their default values:  $\mu_{\text{cr}}= .050$, $\mu_{a} = .085$, $\mu_{\text{im}}$ and $\mu_{\text{me}} = .210$, respectively. Given that the component distributions are Normal, the parameters for sum of conflict resolution and perceptual-motor processing times can be found by summing $\mu$ and $\sigma^2$ parameters as follows:

\begin{align}
\mu_{\rm pm} = 11\mu_{\text{cr}} + \mu_{\text{me}} + \mu_{\text{im}} + 2\mu_{a}
\end{align}

\begin{align}
\sigma_{\rm pm} = \sqrt{11\sigma_{\text{cr}}^2 + \sigma_{\text{me}}^2 + \sigma_{\text{im}}^2 + 2\sigma_{a}^2}
\end{align}

The standard deviation of a Uniform distribution is:

\begin{align}
\sigma = \sqrt{\frac{1}{12}\left(u_\text{max}-u_{min}\right)^2}
\end{align}

## Assumptions

1. Retrieval failures are possible
2. Activation is identical and constant for each chunk except for noise
3. Each data point is independent of others
4. Reaction times follow a Lognormal distribution

# Generate Data

We will use the functions below to generate data for parameter estimation. The `simulate` function accepts the following arguments

- N: the number of data points
- s: the logistic scale parameter
- blc: a constant value added to activation

`simulate` is a high-level function that performs two actions: (1) it creates the model object by calling the function `construct_model`, and (2) draws `N` samples from the resulting model object. The function `construct_model` is responsible for defining the model and has the following arguments:

- $\alpha$: deterministic part of activation
- s: the logistic scale parameter

First, `construct_model` generates the parameters $\mu$ and $\sigma$ for sum of conflict resolution and percpetual-motor processing time. Second, it generates the final model based on a normal distribution object and two lognormal distribution objects.

```julia
function simulate(N; s, blc)
    # construct the model object
    model = construct_model(blc, s)
    # generate N simulated trials
    return rand(model, N)
end

function construct_model(α, s)
  # standard deviation of activation noise in log space
  λ = s * π / sqrt(3)
  # μ and σ parameters for perceptual motor processes
  μ,σ = convolve_normal(motor=(μ=0.21,N=1), cr=(μ=0.05,N=11),visual=(μ=0.085,N=2),
  imaginal=(μ=0.2,N=1))
  # create a model object: perceptual motor processes with two memory retrievals
  model = Normal(μ, σ) + LogNormal(-α, λ) + LogNormal(-α, λ)
  return model
end
```

    construct_model (generic function with 1 method)

In the code block below, 50 simulated reaction times are simulated from the model

```julia
# number of simulated trials
n_trials = 50
# logistic scalar for activation noise
s = 0.3
# base level constant
blc = 1.5
# generate simulated data
data = simulate(n_trials; s, blc);
```

## Define Likelihood Function

The Normal PDF with the shorthand notation $\gamma = \{\mu, \sigma\}$ is,

$f_{\rm Normal}(t; \gamma) = \frac{1}{\sigma \sqrt{2\pi}}e^{- \frac{(t-\mu)^2}{2\sigma^2} }.$

The Lognormal PDF is given by:

$f_{\rm Lognormal}(t; \mu, \sigma) = \frac{1}{t\sigma \sqrt{2\pi}}e^{- \frac{\left(log(t)-\mu\right)^2}{2\sigma^2} }.$

As described in the standard ACT-R tutorial, the task requires the model to count from two to four, meaning there are two memory retrievals---one retrieval to increment from two to three, and one retrieval to increment from three to four. Thus, the likelihood function for the Count model is the convolution of the non-memory retrieval processes (normally distributed) and the two retrieval PDFs (lognormally distributed):

$f(t \mid\text{blc},s) = (f_{{\rm Normal};\gamma_{\rm pm}}* f_{{\rm Lognormal}; \mu, \sigma} * f_{{\rm Lognormal}; \mu, \sigma})(t).$

where $*$ is the convolution operator.

The function `loglike` computes the log likelihood of the data, which has the following arguments:

- blc: the base-level constant parameter
- s: the logistic scalar parameter
- data: a vector of reaction times

`loglike` constructs a model object, convolves the model object and uses the model object to compute the log likelihood of all rts in `data`.

```julia
function loglike(data, blc, s)
  # construct model object
  model = construct_model(blc, s)
  # convolve the component distributions
  convolve!(model)
  # compute the sum log likelihood across all data points
  LL = logpdf.(model, data)
  return sum(LL)
end
```

    loglike (generic function with 1 method)

## Plot likelihood function

In the following code block, we will validate the likelihood function by overlaying it on top of a histogram of simulated data. As you can see, the likelihood provides a close fit to the histogram.

```julia
n_trials = 10_000
s = 0.3
blc = 1.5
times = 1:0.01:3.0
sim_data = simulate(n_trials; s=s, blc=blc);
histogram(sim_data, color=:darkgrey, grid=false, norm=true, leg=false, size=(800,400),
    xlabel="Reaction Time", ylabel="Density")
density = map(x-> loglike(x, blc, s) |> exp, times)
# plot density
plot!(times, density, linewidth=2)
```

## Define Model

The model and prior distributions are summarized as follows:

\begin{align}
\rm blc \sim \rm Normal(1.5,1)
\end{align}

\begin{align}
s \sim \textrm{TNormal}(.3, .5)_{0}^{\infty}
\end{align}

\begin{align}
\rm rt_{i} \sim f(\rm blc, s)
\end{align}

In computer code, the model is specified as follows:

```julia
# prior distributions
priors = (
    blc = (Normal(1.5, 1),),
    s = (Truncated(Normal(0.3, 0.5), 0.0, Inf),)
)
# lower and upper bounds of each parameter
bounds = ((-Inf,Inf),(eps(),Inf))
# define model object
model = DEModel(;priors, model=loglike, data)
# define DE sampler object
de = DE(;priors, bounds, burnin=1000)
```

    DE{Tuple{Tuple{Float64, Float64}, Tuple{Float64, Float64}}, typeof(random_gamma), typeof(DifferentialEvolutionMCMC.Metropolis_Hastings_update!), typeof(compute_posterior!), typeof(sample)}(4, 6, 1000, true, 0.1, 0.1, 0.001, 0.05, 1.0, 0.0, ((-Inf, Inf), (2.220446049250313e-16, Inf)), 0, 1, DifferentialEvolutionMCMC.random_gamma, DifferentialEvolutionMCMC.Metropolis_Hastings_update!, DifferentialEvolutionMCMC.compute_posterior!, StatsBase.sample)

## Estimate Parameters

Now that the priors, likelihood, and Turing model have been specified, we can now estimate the parameters. In the following code, we will run four MCMC chains with the Differential Evolution MCMC (DEMCMC). We will sample for 2,000 iterations and omit the first 1,000 warmup samples.

```julia
n_iter = 2000
chain = sample(model, de, MCMCThreads(), n_iter, progress=true)
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:28

    Chains MCMC chain (1000×4×24 Array{Float64, 3}):

    Iterations        = 1:1:1000
    Number of chains  = 24
    Samples per chain = 1000
    parameters        = blc, s
    internals         = acceptance, lp

    Summary Statistics
      parameters      mean       std   naive_se      mcse         ess      rhat
          Symbol   Float64   Float64    Float64   Float64     Float64   Float64

             blc    1.5170    0.0721     0.0005    0.0015   1968.2946    1.0139
               s    0.3444    0.0419     0.0003    0.0009   2089.9418    1.0096

    Quantiles
      parameters      2.5%     25.0%     50.0%     75.0%     97.5%
          Symbol   Float64   Float64   Float64   Float64   Float64

             blc    1.3828    1.4683    1.5134    1.5622    1.6683
               s    0.2737    0.3146    0.3406    0.3705    0.4367

    ┌ Warning: timestamp of type Missing unknown
    └ @ MCMCChains /home/dfish/.julia/packages/MCMCChains/VwH27/src/chains.jl:364
    ┌ Warning: timestamp of type Missing unknown
    └ @ MCMCChains /home/dfish/.julia/packages/MCMCChains/VwH27/src/chains.jl:364

## Results

The posterior distributions for each parameter are summarized in the output above. Beginning with the trace plot, we see that there are no appearent problems or irregularities. Compared the the previous tutorials that used the NUTS algorithm, autocorrelation is greater for DEMCMC. This is quite normal, as DEMCMC is not as efficient as NUTS. Importantly, however, all of the autocorrelations gradually decrease and none are stuck at a high value for too long. The density plots indicate that the posterior distributions are close to the data generating parameter values for blc and $s$.

```julia
pyplot()
ch = group(chain,:blc)
font_size = 12
p1 = plot(ch, xaxis=font(font_size), yaxis=font(font_size), seriestype=(:traceplot),
  grid=false, size=(250,100), titlefont=font(font_size))
p2 = plot(ch, xaxis=font(font_size), yaxis=font(font_size), seriestype=(:autocorplot),
  grid=false, size=(250,100), titlefont=font(font_size))
p3 = plot(ch, xaxis=font(font_size), yaxis=font(font_size), seriestype=(:mixeddensity),
  grid=false, size=(250,100), titlefont=font(font_size))
pcτ = plot(p1, p2, p3, layout=(3,1), size=(600,600))
```

```julia
ch = group(chain,:s)
p1 = plot(ch, xaxis=font(font_size), yaxis=font(font_size), seriestype=(:traceplot),
  grid=false, size=(250,100), titlefont=font(font_size))
p2 = plot(ch, xaxis=font(font_size), yaxis=font(font_size), seriestype=(:autocorplot),
  grid=false, size=(250,100), titlefont=font(font_size))
p3 = plot(ch, xaxis=font(font_size), yaxis=font(font_size), seriestype=(:mixeddensity),
  grid=false, size=(250,100), titlefont=font(font_size))
pcτ = plot(p1, p2, p3, layout=(3,1), size=(600,600))
```

It is reasonable to pool together chains to generate smoother density plots. This is accomplished by setting the `seriestype` keyword argument to `:pooleddensity`. Alternatively, you can use the function called `pooleddensity`.

```julia
post_plot = plot(chain, xaxis=font(font_size), yaxis=font(font_size), seriestype=(:pooleddensity),
  grid=false, titlefont=font(font_size), size=(600,600), color=:gray, linewidth=2)
```

### Posterior Predictive Distribution

In the code block below, we generate a posterier predictive distribution for the rt distribution.

```julia
preds = posterior_predictive(x->simulate(n_trials; x...), chain, 1000)
rts = vcat(preds...)
post_pred = histogram(rts, xlabel = "RT", ylabel="Frequency", xaxis=font(7), yaxis=font(7),
    grid=false, color=:grey, leg=false, size=(600,300), titlefont=font(7),
    xlims=(.5,3.0))
```

Note that it is possible to pass a summary function in order to generate the posterior predictive distribution of a statistic (e.g. the default summary function is the idenity `x->x`, which returns raw data from the model). In the example below, the function `mean` is passed to summarize the data generated from the model.

```julia
preds = posterior_predictive(x->simulate(n_trials; x...), chain, 1000, mean)
rts = vcat(preds...)
post_pred = histogram(rts, xlabel = "RT", ylabel="Frequency", xaxis=font(7), yaxis=font(7),
    grid=false, color=:grey, leg=false, size=(600,300), titlefont=font(7),
    xlims=(.5,2.5))
```


```python
import os

import pandas as pd
from bokeh.plotting import figure, output_notebook, show
from pystan import StanModel
from sklearn.preprocessing import scale

from rethinking.stan import StanCache
from rethinking.plotting import summaryplot

output_notebook()
```

    Loading BokehJS ...

```python
DATA_FOLDER = '../data'
DIVORCE_FULLPATH = os.path.join(DATA_FOLDER, 'WaffleDivorce.csv')
```

```python
divorce = pd.read_csv(DIVORCE_FULLPATH)
divorce.head()
```

      Location
      Loc
      Population
      MedianAgeMarriage
      Marriage
      Marriage SE
      Divorce
      Divorce SE
      WaffleHouses
      South
      Slaves1860
      Population1860
      PropSlaves1860

      0
      Alabama
      AL
      4.78
      25.3
      20.2
      1.27
      12.7
      0.79
      128
      1
      435080
      964201
      0.45

      1
      Alaska
      AK
      0.71
      25.2
      26.0
      2.93
      12.5
      2.05
      0
      0
      0
      0
      0.00

      2
      Arizona
      AZ
      6.33
      25.8
      20.3
      0.98
      10.8
      0.74
      18
      0
      0
      0
      0.00

      3
      Arkansas
      AR
      2.92
      24.3
      26.4
      1.70
      13.5
      1.22
      41
      1
      111115
      435450
      0.26

      4
      California
      CA
      37.25
      26.8
      19.1
      0.39
      8.0
      0.24
      0
      0
      0
      379994
      0.00

$$
\begin{align*}
    D_i &\sim Normal(\mu_i, \sigma) \\
    \mu_i &= \alpha + \beta_A A_i \\
    \alpha &\sim Normal(10, 10) \\
    \beta_A &\sim Normal(0, 1) \\
    \sigma &\sim Uniform(0, 10)
\end{align*}
$$

```python
m5_1 = StanCache(filename='../models/m5_1.stan', cache_path='../cache').compile()
m5_1.model_code
```

    data {
        int<lower=0> N;
        vector[N] X;
        vector<lower=0, upper=100>[N] divorce_rate;
    }
    parameters {
        real alpha;
        real beta;
        real<lower=0> sigma;
    }
    transformed parameters {
        vector<lower=-100, upper=100>[N] mu;

        mu = alpha + X * beta;
    }
    model {
        alpha ~ normal(10, 10);
        beta ~ normal(0, 1);
        sigma ~ uniform(0, 10);
        divorce_rate ~ normal(mu, sigma);
    }

```python
divorce['MedianAgeMarriage_s'] = scale(
    X=divorce.loc[:, ['MedianAgeMarriage']], with_mean=True, with_std=True)
divorce['Divorce_pct'] = divorce.loc[:, 'Divorce'] / 100
```

```python
divorce_data = dict(
    N=divorce.shape[0],
    K=1,
    X=divorce.MedianAgeMarriage_s,
    divorce_rate=divorce.Divorce
)

fit5_1 = m5_1.sampling(data=divorce_data, iter=500)
print(fit5_1.stansummary(pars=['alpha', 'beta', 'sigma']))
```

    /home/gianluca/.virtualenvs/rethinking/lib/python3.6/site-packages/pystan/misc.py:399: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      elif np.issubdtype(np.asarray(v).dtype, float):

    Inference for Stan model: anon_model_069dd64a1ee6b8cdacde859166ae13ef.
    4 chains, each with iter=500; warmup=250; thin=1;
    post-warmup draws per chain=250, total post-warmup draws=1000.

            mean se_mean     sd   2.5%    25%    50%    75%  97.5%  n_eff   Rhat
    alpha   9.68  6.5e-3   0.21   9.27   9.55   9.68   9.82   10.1   1000    1.0
    beta   -1.02    0.01   0.23  -1.43  -1.19  -1.02  -0.87  -0.54    294    1.0
    sigma   1.52  6.2e-3   0.15   1.24   1.41   1.51   1.61   1.85    607    1.0

    Samples were drawn using NUTS at Mon Mar 12 19:11:26 2018.
    For each parameter, n_eff is a crude measure of effective sample size,
    and Rhat is the potential scale reduction factor on split chains (at
    convergence, Rhat=1).

    /home/gianluca/git/rethinking/src/rethinking/stan.py:95: UserWarning: Pickling fit objects is an experimental feature!
    The relevant StanModel instance must be pickled along with this fit object.
    When unpickling the StanModel must be unpickled first.
      pickle.dump(results, f)

```python
summaryplot(fit5_1, pars=['alpha', 'beta', 'sigma'])
```

```python
divorce['Marriage_s'] = scale(
    X=divorce.loc[:, ['Marriage']], with_mean=True, with_std=True)
```

```python
divorce2_data = dict(
    N=divorce.shape[0],
    K=1,
    X=divorce.Marriage_s,
    divorce_rate=divorce.Divorce
)

fit5_2 = m5_1.sampling(data=divorce2_data, iter=500)
print(fit5_2.stansummary(pars=['alpha', 'beta', 'sigma']))
```

    /home/gianluca/.virtualenvs/rethinking/lib/python3.6/site-packages/pystan/misc.py:399: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      elif np.issubdtype(np.asarray(v).dtype, float):

    Inference for Stan model: anon_model_069dd64a1ee6b8cdacde859166ae13ef.
    4 chains, each with iter=500; warmup=250; thin=1;
    post-warmup draws per chain=250, total post-warmup draws=1000.

            mean se_mean     sd   2.5%    25%    50%    75%  97.5%  n_eff   Rhat
    alpha   9.66    0.01   0.26   9.13    9.5   9.67   9.84  10.18    634   1.01
    beta    0.64  7.3e-3   0.23   0.17   0.47   0.64    0.8   1.09   1000    1.0
    sigma   1.75  8.0e-3   0.17   1.44   1.63   1.74   1.85   2.12    474    1.0

    Samples were drawn using NUTS at Mon Mar 12 19:11:27 2018.
    For each parameter, n_eff is a crude measure of effective sample size,
    and Rhat is the potential scale reduction factor on split chains (at
    convergence, Rhat=1).

    /home/gianluca/git/rethinking/src/rethinking/stan.py:95: UserWarning: Pickling fit objects is an experimental feature!
    The relevant StanModel instance must be pickled along with this fit object.
    When unpickling the StanModel must be unpickled first.
      pickle.dump(results, f)

# Multivariate linear model

$$
\begin{align}
    D_i &\sim Normal(\mu_i, \sigma) \\
    \mu_i &= \alpha + \sum_{j=1}^{n} \beta_i x_{ji} \\
    \alpha &\sim Normal(10, 10) \\
    \beta_R &\sim Normal(0, 1) \\
    \beta_A &\sim Normal(0, 1) \\
    \sigma &\sim Uniform(0, 10)
\end{align}
$$

```python
m5_3 = StanCache(filename='../models/m5_3.stan', cache_path='../cache').compile()
m5_3.model_code
```

    data {
        int<lower=0> N;
        int<lower=0> K;
        matrix[N, K] X;
        vector<lower=0, upper=100>[N] divorce_rate;
    }
    parameters {
        real alpha;
        vector[K] beta;
        real<lower=0> sigma;
    }
    transformed parameters {
        vector<lower=-100, upper=100>[N] mu;

        mu = alpha + X * beta;
    }
    model {
        alpha ~ normal(10, 10);
        beta ~ normal(0, 1);
        sigma ~ uniform(0, 10);
        divorce_rate ~ normal(mu, sigma);
    }

```python
divorce_multi_data = dict(
    N=divorce.shape[0],
    K=2,
    X=divorce.loc[:, ['Marriage_s', 'MedianAgeMarriage_s']],
    divorce_rate = divorce.Divorce
)

fit5_3 = m5_3.sampling(data=divorce_multi_data, iter=500)
print(fit5_3.stansummary(pars=['alpha', 'beta', 'sigma']))
```

    /home/gianluca/.virtualenvs/rethinking/lib/python3.6/site-packages/pystan/misc.py:399: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      elif np.issubdtype(np.asarray(v).dtype, float):

    Inference for Stan model: anon_model_623c071ba8c404bae8e2868021df5aa8.
    4 chains, each with iter=500; warmup=250; thin=1;
    post-warmup draws per chain=250, total post-warmup draws=1000.

              mean se_mean     sd   2.5%    25%    50%    75%  97.5%  n_eff   Rhat
    alpha     9.45    0.29   1.55   6.85   9.54   9.69   9.83  10.14     28   1.11
    beta[0]  -0.15    0.02   0.34  -0.86  -0.35  -0.14   0.05   0.51    251   1.01
    beta[1]  -1.09    0.05   0.38  -1.68  -1.32  -1.12  -0.92   -0.3     56   1.04
    sigma     1.74    0.26   1.28   1.24   1.41   1.52   1.66   4.51     25   1.12

    Samples were drawn using NUTS at Mon Mar  5 22:10:13 2018.
    For each parameter, n_eff is a crude measure of effective sample size,
    and Rhat is the potential scale reduction factor on split chains (at
    convergence, Rhat=1).

```python
summaryplot(fit5_3, pars=['alpha', 'beta', 'sigma'])
```

# Predictor residual plots

```python

```


# Survey Analysis Report
### Team Ford
Samantha Howard, Jyotsna Singh, Ishaan Pathak and Nick Mouaikel

# Sam's Analysis

## Abstract
Short summary about the data and what it tells you about your project.

## Data input
In this section include code that reads in the csv file

```python
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import seaborn as sns
```

```python
dist = pd.read_csv("2022_Project_distance_Matrix.csv")
dist.head()
```

      ARFL
      Argonne
      Boeing
      Delta Dental
      Ford
      Hope Village
      Kellogg's
      Neogen
      Old Nation
      Qside

      0
      7
      4
      9
      8
      3
      0
      6
      2
      5
      1

      1
      0
      1
      3
      2
      4
      8
      6
      5
      7
      9

      2
      3
      2
      1
      6
      0
      8
      5
      4
      7
      9

      3
      8
      7
      3
      9
      4
      5
      0
      6
      1
      2

      4
      0
      2
      1
      8
      7
      3
      9
      6
      4
      5

```python
# Team List
Teams =["ARFL","Argonne","Boeing","Delta Dental","Ford","Hope Village","Kellogg's","Neogen","Old Nation","Qside"]

# Broken up by team
#team = dist[dist['team'] == 0]
ARFL = dist[dist['ARFL'] == 0]
Argonne = dist[dist['Argonne'] == 0]

Boeing = dist[dist['Boeing'] == 0]
Delta = dist[dist['Delta Dental'] == 0]

Ford = dist[dist['Ford'] == 0] #6 members
Hope = dist[dist['Hope Village'] == 0]

Kellogg = dist[dist["Kellogg's"] == 0]
Neogen = dist[dist['Neogen'] == 0]

Old = dist[dist['Old Nation'] == 0]
Qside = dist[dist['Qside'] == 0] #2 members
```

## Data Cleaning
In this section provide code for converting the raw data into clean and usable data structures (if needed)

While there is no NA values there is a potental error in the collection of the data. As the team Qside had 2 entries, and Team ford had 6. And these are not the correct number of members for either.

```python
#no missing entries
dist.isna().sum()
```

    ARFL            0
    Argonne         0
    Boeing          0
    Delta Dental    0
    Ford            0
    Hope Village    0
    Kellogg's       0
    Neogen          0
    Old Nation      0
    Qside           0
    dtype: int64

## Data Modeling

```python
corr = dist.corr()
corr
```

      ARFL
      Argonne
      Boeing
      Delta Dental
      Ford
      Hope Village
      Kellogg's
      Neogen
      Old Nation
      Qside

      ARFL
      1.000000
      0.316525
      0.361881
      0.243643
      0.061635
      -0.577136
      -0.211970
      -0.017531
      -0.474939
      -0.494224

      Argonne
      0.316525
      1.000000
      0.483512
      0.063318
      0.157784
      -0.434898
      -0.338808
      -0.141758
      -0.436384
      -0.557939

      Boeing
      0.361881
      0.483512
      1.000000
      -0.146465
      0.183945
      -0.707230
      -0.101455
      0.156961
      -0.505099
      -0.659948

      Delta Dental
      0.243643
      0.063318
      -0.146465
      1.000000
      0.042934
      -0.128369
      -0.424887
      -0.205687
      -0.154399
      -0.202978

      Ford
      0.061635
      0.157784
      0.183945
      0.042934
      1.000000
      -0.410659
      -0.120666
      0.142023
      -0.591787
      -0.548476

      Hope Village
      -0.577136
      -0.434898
      -0.707230
      -0.128369
      -0.410659
      1.000000
      -0.041932
      -0.293983
      0.681481
      0.728218

      Kellogg's
      -0.211970
      -0.338808
      -0.101455
      -0.424887
      -0.120666
      -0.041932
      1.000000
      0.045035
      -0.052038
      0.211330

      Neogen
      -0.017531
      -0.141758
      0.156961
      -0.205687
      0.142023
      -0.293983
      0.045035
      1.000000
      -0.201630
      -0.261360

      Old Nation
      -0.474939
      -0.436384
      -0.505099
      -0.154399
      -0.591787
      0.681481
      -0.052038
      -0.201630
      1.000000
      0.635497

      Qside
      -0.494224
      -0.557939
      -0.659948
      -0.202978
      -0.548476
      0.728218
      0.211330
      -0.261360
      0.635497
      1.000000

```python
# bad, should change the groupby. Many "NA" values
summary = dist.groupby(Teams).size().unstack()

from scipy.spatial.distance import squareform
from scipy.spatial.distance import pdist

pairwise = pd.DataFrame(
    squareform(pdist(summary)),
    columns = summary.index,
    index = summary.index)
#pairwise
```

## Data Visualization

This section make some graphs visualizing your results.  A distance matrix and/or network graph may be cool.  Think though the best way to show what you learned.

Here using Seaborn, it isn't isn't a distance matrix but I thought the visualization could be intresting. I was courious if it could just give a quick glance at the general likness of the projects to another. I would not recommend this model.

The easist way to understand this visulization is the paler the color, the more similar the two groups are. The Darker the blue/red is is the more differnet they are.

```python
mask = np.triu(np.ones_like(corr, dtype=bool))
cmap = sns.diverging_palette(230, 20, as_cmap=True)
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})
#closer to white is more similar to each other. Stronger color is less similar.
```

## Conclusion

This should be similar to the abstract but with more details. What can you conclude about your project from this data?

## Sources
[Pairwise Distance Matrix Tutoral](https://drawingfromdata.com/pandas/clustering/making-a-pairwise-distance-matrix-in-pandas.html)

[Seaborn Correlation](https://seaborn.pydata.org/examples/many_pairwise_correlations.html)

# Jyotsna's Analysis

```python
data = pd.read_csv("2022_Project_distance_Matrix.csv")
```

## Disagrement

The heatmap below shows how much the people on a team agreed or disagreed on which projects were similar to their own. The darker colors mean higher disagreemnent and lighter colors mean lower disagreemnet.

```python
temp = {}

for a in data.columns.values:
    df = data[data[a] == 0]
    #temp[a] = np.array(df.sum(0)/df.shape[0])
    temp[a] = df.std(axis=0)

prob_matrix = pd.DataFrame(temp).T.fillna(0)

plt.figure(figsize=(12,10))
sns.heatmap(prob_matrix, cmap="Blues")
plt.title("DISAGREEMENT HEATMAP", fontsize = 20);
plt.xticks(fontsize=12);
plt.yticks(fontsize=12, rotation=360);
```

# Ishaan's Analysis

```python
import numpy as np
import scipy as sc
from scipy.spatial import distance_matrix
import pandas as pd
import seaborn as sns
import matplotlib as mpl
import matplotlib.pyplot as plt
from scipy.spatial.distance import pdist
from scipy.spatial.distance import squareform
```

```python
data = pd.read_csv('2022_Project_distance_Matrix.csv')
ARFL = data[data['ARFL'] == 0]
ARFL
```

      ARFL
      Argonne
      Boeing
      Delta Dental
      Ford
      Hope Village
      Kellogg's
      Neogen
      Old Nation
      Qside

      1
      0
      1
      3
      2
      4
      8
      6
      5
      7
      9

      4
      0
      2
      1
      8
      7
      3
      9
      6
      4
      5

      14
      0
      4
      1
      7
      2
      8
      5
      3
      6
      9

      18
      0
      5
      9
      2
      1
      3
      8
      4
      7
      6

      22
      0
      3
      1
      7
      6
      9
      5
      2
      4
      8

```python
team_wise_data = pd.DataFrame(data=[], columns=data.columns)
for i in data.columns:
    temp_team = data[data[i] == 0]
    team_wise_data = pd.concat([team_wise_data, temp_team])
```

```python
team_wise_data.head()
```

      ARFL
      Argonne
      Boeing
      Delta Dental
      Ford
      Hope Village
      Kellogg's
      Neogen
      Old Nation
      Qside

      1
      0
      1
      3
      2
      4
      8
      6
      5
      7
      9

      4
      0
      2
      1
      8
      7
      3
      9
      6
      4
      5

      14
      0
      4
      1
      7
      2
      8
      5
      3
      6
      9

      18
      0
      5
      9
      2
      1
      3
      8
      4
      7
      6

      22
      0
      3
      1
      7
      6
      9
      5
      2
      4
      8

```python
pairwise_top = pd.DataFrame(squareform(pdist(team_wise_data, metric='cosine')),
                            columns = team_wise_data.index,
                            index = team_wise_data.index)

plt.figure(figsize=(15,15))
sns.heatmap(pairwise_top, cmap='OrRd', linewidth=1)
```

# Nick's Analysis

```python
import pandas as pd
import numpy as np
from scipy.spatial import distance_matrix
import sympy as sym
import seaborn as sns
import matplotlib.pyplot as plt
```

```python
data = pd.read_csv('2022_Project_distance_Matrix.csv',header='infer')
data.head()
```

```python
dist_mat = distance_matrix(data.T,data.T,p=2)
fig, ax = plt.subplots(figsize = (15,15))
ax = sns.heatmap(dist_mat,annot=True)
```

```python
np.dot(data.T,data)
```


# Solving Stiff Equations
### Chris Rackauckas

This tutorial is for getting into the extra features for solving stiff ordinary
differential equations in an efficient manner. Solving stiff ordinary
differential equations requires specializing the linear solver on properties of
the Jacobian in order to cut down on the O(n^3) linear solve and the O(n^2)
back-solves. Note that these same functions and controls also extend to stiff
SDEs, DDEs, DAEs, etc.

## Code Optimization for Differential Equations

### Writing Efficient Code

For a detailed tutorial on how to optimize one's DifferentialEquations.jl code,
please see the
[Optimizing DiffEq Code tutorial](http://tutorials.juliadiffeq.org/html/introduction/03-optimizing_diffeq_code.html).

### Choosing a Good Solver

Choosing a good solver is required for getting top notch speed. General
recommendations can be found on the solver page (for example, the
[ODE Solver Recommendations](http://docs.juliadiffeq.org/latest/solvers/ode_solve.html)).
The current recommendations can be simplified to a Rosenbrock method
(`Rosenbrock23` or `Rodas5`) for smaller (<50 ODEs) problems, ESDIRK methods
for slightly larger (`TRBDF2` or `KenCarp4` for <2000 ODEs), and Sundials
`CVODE_BDF` for even larger problems. `lsoda` from
[LSODA.jl](https://github.com/rveltz/LSODA.jl) is generally worth a try.

More details on the solver to choose can be found by benchmarking. See the
[DiffEqBenchmarks](https://github.com/JuliaDiffEq/DiffEqBenchmarks.jl) to
compare many solvers on many problems.

### Check Out the Speed FAQ

See [this FAQ](http://docs.juliadiffeq.org/latest/basics/faq.html#Performance-1)
for information on common pitfalls and how to improve performance.

### Setting Up Your Julia Installation for Speed

Julia uses an underlying BLAS implementation for its matrix multiplications
and factorizations. This library is automatically multithreaded and accelerates
the internal linear algebra of DifferentialEquations.jl. However, for optimality,
you should make sure that the number of BLAS threads that you are using matches
the number of physical cores and not the number of logical cores. See
[this issue for more details](https://github.com/JuliaLang/julia/issues/33409).

To check the number of BLAS threads, use:

```julia
ccall((:openblas_get_num_threads64_, Base.libblas_name), Cint, ())
```

If I want to set this directly to 4 threads, I would use:

```julia
using LinearAlgebra
LinearAlgebra.BLAS.set_num_threads(4)
```

Additionally, in some cases Intel's MKL might be a faster BLAS than the standard
BLAS that ships with Julia (OpenBLAS). To switch your BLAS implementation, you
can use [MKL.jl](https://github.com/JuliaComputing/MKL.jl) which will accelerate
the linear algebra routines. Please see the package for the limitations.

### Use Accelerator Hardware

When possible, use GPUs. If your ODE system is small and you need to solve it
with very many different parameters, see the
[ensembles interface](http://docs.juliadiffeq.org/latest/features/ensemble.html)
and [DiffEqGPU.jl](https://github.com/JuliaDiffEq/DiffEqGPU.jl). If your problem
is large, consider using a [CuArray](https://github.com/JuliaGPU/CuArrays.jl)
for the state to allow for GPU-parallelism of the internal linear algebra.

## Speeding Up Jacobian Calculations

When one is using an implicit or semi-implicit differential equation solver,
the Jacobian must be built at many iterations and this can be one of the most
expensive steps. There are two pieces that must be optimized in order to reach
maximal efficiency when solving stiff equations: the sparsity pattern and the
construction of the Jacobian. The construction is filling the matrix
`J` with values, while the sparsity pattern is what `J` to use.

The sparsity pattern is given by a prototype matrix, the `jac_prototype`, which
will be copied to be used as `J`. The default is for `J` to be a `Matrix`,
i.e. a dense matrix. However, if you know the sparsity of your problem, then
you can pass a different matrix type. For example, a `SparseMatrixCSC` will
give a sparse matrix. Additionally, structured matrix types like `Tridiagonal`,
`BandedMatrix` (from
[BandedMatrices.jl](https://github.com/JuliaMatrices/BandedMatrices.jl)),
`BlockBandedMatrix` (from
[BlockBandedMatrices.jl](https://github.com/JuliaMatrices/BlockBandedMatrices.jl)),
and more can be given. DifferentialEquations.jl will internally use this matrix
type, making the factorizations faster by utilizing the specialized forms.

For the construction, there are 3 ways to fill `J`:

- The default, which uses normal finite/automatic differentiation
- A function `jac(J,u,p,t)` which directly computes the values of `J`
- A `colorvec` which defines a sparse differentiation scheme.

We will now showcase how to make use of this functionality with growing complexity.

### Declaring Jacobian Functions

Let's solve the Rosenbrock equations:

$$\begin{align}
dy_1 &= -0.04y₁ + 10^4 y_2 y_3 \\
dy_2 &= 0.04 y_1 - 10^4 y_2 y_3 - 3*10^7 y_{2}^2 \\
dy_3 &= 3*10^7 y_{3}^2 \\
\end{align}$$

In order to reduce the Jacobian construction cost, one can describe a Jacobian
function by using the `jac` argument for the `ODEFunction`. First, let's do
a standard `ODEProblem`:

```julia
using DifferentialEquations
function rober(du,u,p,t)
  y₁,y₂,y₃ = u
  k₁,k₂,k₃ = p
  du[1] = -k₁*y₁+k₃*y₂*y₃
  du[2] =  k₁*y₁-k₂*y₂^2-k₃*y₂*y₃
  du[3] =  k₂*y₂^2
  nothing
end
prob = ODEProblem(rober,[1.0,0.0,0.0],(0.0,1e5),(0.04,3e7,1e4))
sol = solve(prob,Rosenbrock23())

using Plots
plot(sol, xscale=:log10, tspan=(1e-6, 1e5), layout=(3,1))
```

```julia
using BenchmarkTools
@btime solve(prob)
```

Now we want to add the Jacobian. First we have to derive the Jacobian
$\frac{df_i}{du_j}$ which is `J[i,j]`. From this we get:

```julia
function rober_jac(J,u,p,t)
  y₁,y₂,y₃ = u
  k₁,k₂,k₃ = p
  J[1,1] = k₁ * -1
  J[2,1] = k₁
  J[3,1] = 0
  J[1,2] = y₃ * k₃
  J[2,2] = y₂ * k₂ * -2 + y₃ * k₃ * -1
  J[3,2] = y₂ * 2 * k₂
  J[1,3] = k₃ * y₂
  J[2,3] = k₃ * y₂ * -1
  J[3,3] = 0
  nothing
end
f = ODEFunction(rober, jac=rober_jac)
prob_jac = ODEProblem(f,[1.0,0.0,0.0],(0.0,1e5),(0.04,3e7,1e4))

@btime solve(prob_jac)
```

### Automatic Derivation of Jacobian Functions

But that was hard! If you want to take the symbolic Jacobian of numerical
code, we can make use of [ModelingToolkit.jl](https://github.com/JuliaDiffEq/ModelingToolkit.jl)
to symbolicify the numerical code and do the symbolic calculation and return
the Julia code for this.

```julia
using ModelingToolkit
de = modelingtoolkitize(prob)
ModelingToolkit.generate_jacobian(de...)[2] # Second is in-place
```

which outputs:

```julia
:((##MTIIPVar#376, u, p, t)->begin
          #= C:\Users\accou\.julia\packages\ModelingToolkit\czHtj\src\utils.jl:65 =#
          #= C:\Users\accou\.julia\packages\ModelingToolkit\czHtj\src\utils.jl:66 =#
          let (x₁, x₂, x₃, α₁, α₂, α₃) = (u[1], u[2], u[3], p[1], p[2], p[3])
              ##MTIIPVar#376[1] = α₁ * -1
              ##MTIIPVar#376[2] = α₁
              ##MTIIPVar#376[3] = 0
              ##MTIIPVar#376[4] = x₃ * α₃
              ##MTIIPVar#376[5] = x₂ * α₂ * -2 + x₃ * α₃ * -1
              ##MTIIPVar#376[6] = x₂ * 2 * α₂
              ##MTIIPVar#376[7] = α₃ * x₂
              ##MTIIPVar#376[8] = α₃ * x₂ * -1
              ##MTIIPVar#376[9] = 0
          end
          #= C:\Users\accou\.julia\packages\ModelingToolkit\czHtj\src\utils.jl:67 =#
          nothing
      end)
```

Now let's use that to give the analytical solution Jacobian:

```julia
jac = eval(ModelingToolkit.generate_jacobian(de...)[2])
f = ODEFunction(rober, jac=jac)
prob_jac = ODEProblem(f,[1.0,0.0,0.0],(0.0,1e5),(0.04,3e7,1e4))
```

### Declaring a Sparse Jacobian

Jacobian sparsity is declared by the `jac_prototype` argument in the `ODEFunction`.
Note that you should only do this if the sparsity is high, for example, 0.1%
of the matrix is non-zeros, otherwise the overhead of sparse matrices can be higher
than the gains from sparse differentiation!

But as a demonstration, let's build a sparse matrix for the Rober problem. We
can do this by gathering the `I` and `J` pairs for the non-zero components, like:

```julia
I = [1,2,1,2,3,1,2]
J = [1,1,2,2,2,3,3]
using SparseArrays
jac_prototype = sparse(I,J,1.0)
```

Now this is the sparse matrix prototype that we want to use in our solver, which
we then pass like:

```julia
f = ODEFunction(rober, jac=jac, jac_prototype=jac_prototype)
prob_jac = ODEProblem(f,[1.0,0.0,0.0],(0.0,1e5),(0.04,3e7,1e4))
```

### Automatic Sparsity Detection

One of the useful companion tools for DifferentialEquations.jl is
[SparsityDetection.jl](https://github.com/JuliaDiffEq/SparsityDetection.jl).
This allows for automatic declaration of Jacobian sparsity types. To see this
in action, let's look at the 2-dimensional Brusselator equation:

```julia
const N = 32
const xyd_brusselator = range(0,stop=1,length=N)
brusselator_f(x, y, t) = (((x-0.3)^2 + (y-0.6)^2) <= 0.1^2) * (t >= 1.1) * 5.
limit(a, N) = a == N+1 ? 1 : a == 0 ? N : a
function brusselator_2d_loop(du, u, p, t)
  A, B, alpha, dx = p
  alpha = alpha/dx^2
  @inbounds for I in CartesianIndices((N, N))
    i, j = Tuple(I)
    x, y = xyd_brusselator[I[1]], xyd_brusselator[I[2]]
    ip1, im1, jp1, jm1 = limit(i+1, N), limit(i-1, N), limit(j+1, N), limit(j-1, N)
    du[i,j,1] = alpha*(u[im1,j,1] + u[ip1,j,1] + u[i,jp1,1] + u[i,jm1,1] - 4u[i,j,1]) +
                B + u[i,j,1]^2*u[i,j,2] - (A + 1)*u[i,j,1] + brusselator_f(x, y, t)
    du[i,j,2] = alpha*(u[im1,j,2] + u[ip1,j,2] + u[i,jp1,2] + u[i,jm1,2] - 4u[i,j,2]) +
                A*u[i,j,1] - u[i,j,1]^2*u[i,j,2]
    end
end
p = (3.4, 1., 10., step(xyd_brusselator))
```

Given this setup, we can give and example `input` and `output` and call `sparsity!`
on our function with the example arguments and it will kick out a sparse matrix
with our pattern, that we can turn into our `jac_prototype`.

```julia
using SparsityDetection, SparseArrays
input = rand(32,32,2)
output = similar(input)
sparsity_pattern = sparsity!(brusselator_2d_loop,output,input,p,0.0)
jac_sparsity = Float64.(sparse(sparsity_pattern))
```

Let's double check what our sparsity pattern looks like:

```julia
using Plots
spy(jac_sparsity,markersize=1,colorbar=false,color=:deep)
```

That's neat, and would be tedius to build by hand! Now we just pass it to the
`ODEFunction` like as before:

```julia
f = ODEFunction(brusselator_2d_loop;jac_prototype=jac_sparsity)
```

Build the `ODEProblem`:

```julia
function init_brusselator_2d(xyd)
  N = length(xyd)
  u = zeros(N, N, 2)
  for I in CartesianIndices((N, N))
    x = xyd[I[1]]
    y = xyd[I[2]]
    u[I,1] = 22*(y*(1-y))^(3/2)
    u[I,2] = 27*(x*(1-x))^(3/2)
  end
  u
end
u0 = init_brusselator_2d(xyd_brusselator)
prob_ode_brusselator_2d = ODEProblem(brusselator_2d_loop,
                                     u0,(0.,11.5),p)

prob_ode_brusselator_2d_sparse = ODEProblem(f,
                                     u0,(0.,11.5),p)
```

Now let's see how the version with sparsity compares to the version without:

```julia
@btime solve(prob_ode_brusselator_2d,save_everystep=false)
@btime solve(prob_ode_brusselator_2d_sparse,save_everystep=false)
```

### Declaring Color Vectors for Fast Construction

If you cannot directly define a Jacobian function, you can use the `colorvec`
to speed up the Jacobian construction. What the `colorvec` does is allows for
calculating multiple columns of a Jacobian simultaniously by using the sparsity
pattern. An explanation of matrix coloring can be found in the
[MIT 18.337 Lecture Notes](https://mitmath.github.io/18337/lecture9/stiff_odes).

To perform general matrix coloring, we can use
[SparseDiffTools.jl](https://github.com/JuliaDiffEq/SparseDiffTools.jl). For
example, for the Brusselator equation:

```julia
using SparseDiffTools
colorvec = matrix_colors(jac_sparsity)
@show maximum(colorvec)
```

This means that we can now calculate the Jacobian in 12 function calls. This is
a nice reduction from 2048 using only automated tooling! To now make use of this
inside of the ODE solver, you simply need to declare the colorvec:

```julia
f = ODEFunction(brusselator_2d_loop;jac_prototype=jac_sparsity,
                                    colorvec=colorvec)
prob_ode_brusselator_2d_sparse = ODEProblem(f,
                                     init_brusselator_2d(xyd_brusselator),
                                     (0.,11.5),p)
@btime solve(prob_ode_brusselator_2d_sparse,save_everystep=false)
```

Notice the massive speed enhancement!

## Defining Linear Solver Routines and Jacobian-Free Newton-Krylov

A completely different way to optimize the linear solvers for large sparse
matrices is to use a Krylov subpsace method. This requires choosing a linear
solver for changing to a Krylov method. Optionally, one can use a Jacobian-free
operator to reduce the memory requirements.

### Declaring a Jacobian-Free Newton-Krylov Implementation

To swap the linear solver out, we use the `linsolve` command and choose the
GMRES linear solver.

```julia
@btime solve(prob_ode_brusselator_2d,TRBDF2(linsolve=LinSolveGMRES()),save_everystep=false)
@btime solve(prob_ode_brusselator_2d_sparse,TRBDF2(linsolve=LinSolveGMRES()),save_everystep=false)
```

For more information on linear solver choices, see the
[linear solver documentation](http://docs.juliadiffeq.org/latest/features/linear_nonlinear.html).

On this problem, handling the sparsity correctly seemed to give much more of a
speedup than going to a Krylov approach, but that can be dependent on the problem
(and whether a good preconditioner is found).

We can also enhance this by using a Jacobian-Free implementation of `f'(x)*v`.
To define the Jacobian-Free operator, we can use
[DiffEqOperators.jl](https://github.com/JuliaDiffEq/DiffEqOperators.jl) to generate
an operator `JacVecOperator` such that `Jv*v` performs `f'(x)*v` without building
the Jacobian matrix.

```julia
using DiffEqOperators
Jv = JacVecOperator(brusselator_2d_loop,u0,p,0.0)
```

and then we can use this by making it our `jac_prototype`:

```julia
f = ODEFunction(brusselator_2d_loop;jac_prototype=Jv)
prob_ode_brusselator_2d_jacfree = ODEProblem(f,u0,(0.,11.5),p)
@btime solve(prob_ode_brusselator_2d_jacfree,TRBDF2(linsolve=LinSolveGMRES()),save_everystep=false)
```

### Adding a Preconditioner

The [linear solver documentation](http://docs.juliadiffeq.org/latest/features/linear_nonlinear.html#IterativeSolvers.jl-Based-Methods-1)
shows how you can add a preconditioner to the GMRES. For example, you can
use packages like [AlgebraicMultigrid.jl](https://github.com/JuliaLinearAlgebra/AlgebraicMultigrid.jl)
to add an algebraic multigrid (AMG) or [IncompleteLU.jl](https://github.com/haampie/IncompleteLU.jl)
for an incomplete LU-factorization (iLU).

```julia
using AlgebraicMultigrid
pc = aspreconditioner(ruge_stuben(jac_sparsity))
@btime solve(prob_ode_brusselator_2d_jacfree,TRBDF2(linsolve=LinSolveGMRES(Pl=pc)),save_everystep=false)
```

## Using Structured Matrix Types

If your sparsity pattern follows a specific structure, for example a banded
matrix, then you can declare `jac_prototype` to be of that structure and then
additional optimizations will come for free. Note that in this case, it is
not necessary to provide a `colorvec` since the color vector will be analytically
derived from the structure of the matrix.

The matrices which are allowed are those which satisfy the
[ArrayInterface.jl](https://github.com/JuliaDiffEq/ArrayInterface.jl) interface
for automatically-colorable matrices. These include:

- Bidiagonal
- Tridiagonal
- SymTridiagonal
- BandedMatrix ([BandedMatrices.jl](https://github.com/JuliaMatrices/BandedMatrices.jl))
- BlockBandedMatrix ([BlockBandedMatrices.jl](https://github.com/JuliaMatrices/BlockBandedMatrices.jl))

Matrices which do not satisfy this interface can still be used, but the matrix
coloring will not be automatic, and an appropriate linear solver may need to
be given (otherwise it will default to attempting an LU-decomposition).

## Sundials-Specific Handling

While much of the setup makes the transition to using Sundials automatic, there
are some differences between the pure Julia implementations and the Sundials
implementations which must be taken note of. These are all detailed in the
[Sundials solver documentation](http://docs.juliadiffeq.org/latest/solvers/ode_solve.html#Sundials.jl-1),
but here we will highlight the main details which one should make note of.

Defining a sparse matrix and a Jacobian for Sundials works just like any other
package. The core difference is in the choice of the linear solver. With Sundials,
the linear solver choice is done with a Symbol in the `linear_solver` from a
preset list. Particular choices of note are `:Band` for a banded matrix and
`:GMRES` for using GMRES. If you are using Sundials, `:GMRES` will not require
defining the JacVecOperator, and instead will always make use of a Jacobian-Free
Newton Krylov (with numerical differentiation). Thus on this problem we could do:

```julia
using Sundials
# Sparse Version
@btime solve(prob_ode_brusselator_2d_sparse,CVODE_BDF(),save_everystep=false)
# GMRES Version: Doesn't require any extra stuff!
@btime solve(prob_ode_brusselator_2d,CVODE_BDF(linear_solver=:GMRES),save_everystep=false)
```

Details for setting up a preconditioner with Sundials can be found at the
[Sundials solver page](http://docs.juliadiffeq.org/latest/solvers/ode_solve.html#Sundials.jl-1).

## Handling Mass Matrices

Instead of just defining an ODE as $u' = f(u,p,t)$, it can be common to express
the differential equation in the form with a mass matrix:

$$Mu' = f(u,p,t)$$

where $M$ is known as the mass matrix. Let's solve the Robertson equation.
At the top we wrote this equation as:

$$\begin{align}
dy_1 &= -0.04y₁ + 10^4 y_2 y_3 \\
dy_2 &= 0.04 y_1 - 10^4 y_2 y_3 - 3*10^7 y_{2}^2 \\
dy_3 &= 3*10^7 y_{3}^2 \\
\end{align}$$

But we can instead write this with a conservation relation:

$$\begin{align}
dy_1 &= -0.04y₁ + 10^4 y_2 y_3 \\
dy_2 &= 0.04 y_1 - 10^4 y_2 y_3 - 3*10^7 y_{2}^2 \\
1 &=  y_{1} + y_{2} + y_{3} \\
\end{align}$$

In this form, we can write this as a mass matrix ODE where $M$ is singular
(this is another form of a differential-algebraic equation (DAE)). Here, the
last row of `M` is just zero. We can implement this form as:

```julia
using DifferentialEquations
function rober(du,u,p,t)
  y₁,y₂,y₃ = u
  k₁,k₂,k₃ = p
  du[1] = -k₁*y₁+k₃*y₂*y₃
  du[2] =  k₁*y₁-k₂*y₂^2-k₃*y₂*y₃
  du[3] =  y₁ + y₂ + y₃ - 1
  nothing
end
M = [1. 0  0
     0  1. 0
     0  0  0]
f = ODEFunction(rober,mass_matrix=M)
prob_mm = ODEProblem(f,[1.0,0.0,0.0],(0.0,1e5),(0.04,3e7,1e4))
sol = solve(prob_mm,Rodas5())

plot(sol, xscale=:log10, tspan=(1e-6, 1e5), layout=(3,1))
```

Note that if your mass matrix is singular, i.e. your system is a DAE, then you
need to make sure you choose
[a solver that is compatible with DAEs](http://docs.juliadiffeq.org/latest/solvers/dae_solve.html#Full-List-of-Methods-1)


# `GiRaFFE_NRPy`: Solving the Induction Equation

## Author: Patrick Nelson

This notebook documents the function from the original `GiRaFFE` that calculates the flux for $A_i$ according to the method of Harten, Lax, von Leer, and Einfeldt (HLLE), assuming that we have calculated the values of the velocity and magnetic field on the cell faces according to the piecewise-parabolic method (PPM) of [Colella and Woodward (1984)](https://crd.lbl.gov/assets/pubs_presos/AMCS/ANAG/A141984.pdf), modified for the case of GRFFE.

**Notebook Status:**  Validated

**Validation Notes:** This code has been validated by showing that it converges to the exact answer at the expected order

### NRPy+ Source Code for this module:
* [GiRaFFE_NRPy/Afield_flux.py](../../edit/in_progress/GiRaFFE_NRPy/Afield_flux.py)

Our goal in this module is to write the code necessary to solve the induction equation
$$
\partial_t A_i = \underbrace{\epsilon_{ijk} v^j B^k}_{\rm Flux\ terms} - \underbrace{\partial_i \left(\alpha \Phi - \beta^j A_j \right)}_{\rm Gauge\ terms}.
$$
To properly handle the flux terms and avoiding problems with shocks, we cannot simply take a cross product of the velocity and magnetic field at the cell centers. Instead, we must solve the Riemann problem at the cell faces using the reconstructed values of the velocity and magnetic field on either side of the cell faces. The reconstruction is done using PPM (see [here](Tutorial-GiRaFFE_NRPy-PPM.ipynb)); in this module, we will assume that that step has already been done. Metric quantities are assumed to have been interpolated to cell faces, as is done in [this](Tutorial-GiRaFFE_NRPy-Metric_Face_Values.ipynb) tutorial.

Tóth's [paper](https://www.sciencedirect.com/science/article/pii/S0021999100965197?via%3Dihub), Eqs. 30 and 31, are one of the first implementations of such a scheme. The original GiRaFFE used a 2D version of the algorithm from [Del Zanna, et al. (2002)](https://arxiv.org/abs/astro-ph/0210618); but since we are not using staggered grids, we can greatly simplify this algorithm with respect to the version used in the original `GiRaFFE`. Instead, we will adapt the implementations of the algorithm used in [Mewes, et al. (2020)](https://arxiv.org/abs/2002.06225) and [Giacomazzo, et al. (2011)](https://arxiv.org/abs/1009.2468), Eqs. 3-11.

We first write the flux contribution to the induction equation RHS as
$$
\partial_t A_i = -E_i,
$$
where the electric field $E_i$ is given in ideal MHD (of which FFE is a subset) as
$$
-E_i = \epsilon_{ijk} v^j B^k,
$$
where $v^i$ is the drift velocity,  $B^i$ is the magnetic field, and $\epsilon_{ijk} = \sqrt{\gamma} [ijk]$ is the Levi-Civita tensor.
In Cartesian coordinates,
\begin{align}
-E_x &= -F^z(B^y) = F^y(B^z) \\
-E_y &= F^z(B^x) = -F^x(B^z) \\
-E_z &= -F^y(B^x) = F^x(B^y), \\
\end{align}
where
$$
F^i(B^j) = \sqrt{\gamma} (v^i B^j - v^j B^i).
$$
To compute the actual contribution to the RHS in some direction $i$, we average the above listed field as calculated on the $+j$, $-j$, $+k$, and $-k$ faces. That is, at some point $(i,j,k)$ on the grid,
\begin{align}
-E_x(x_i,y_j,z_k) &= \frac{1}{4} \left( F_{\rm HLL}^y(B^z)_{(i,j+1/2,k)}+F_{\rm HLL}^y(B^z)_{(i,j-1/2,k)}-F_{\rm HLL}^z(B^y)_{(i,j,k+1/2)}-F_{\rm HLL}^z(B^y)_{(i,j,k-1/2)} \right) \\
-E_y(x_i,y_j,z_k) &= \frac{1}{4} \left( -F_{\rm HLL}^x(B^z)_{(i+1/2,j,k)}-F_{\rm HLL}^x(B^z)_{(i-1/2,j,k)}+F_{\rm HLL}^z(B^x)_{(i,j,k+1/2)}+F_{\rm HLL}^z(B^x)_{(i,j,k-1/2)} \right) \\
-E_x(x_i,y_j,z_k) &= \frac{1}{4} \left( F_{\rm HLL}^x(B^y)_{(i+1/2,j,k)}+F_{\rm HLL}^x(B^y)_{(i-1/2,j,k)}-F_{\rm HLL}^y(B^x)_{(i,j+1/2,k)}-F_{\rm HLL}^y(B^x)_{(i,j-1/2,k)} \right). \\
\end{align}
Note the use of $F_{\rm HLL}$ here. This change signifies that the quantity output here is from the HLLE Riemann solver. Note also the indices on the fluxes. Values of $\pm 1/2$ indicate that these are computed on cell faces using the reconstructed values of $v^i$ and $B^i$ and the interpolated values of the metric gridfunctions. So,
$$
F_{\rm HLL}^i(B^j) = \frac{c_{\rm min} F_{\rm R}^i(B^j) + c_{\rm max} F_{\rm L}^i(B^j) - c_{\rm min} c_{\rm max} (B_{\rm R}^j-B_{\rm L}^j)}{c_{\rm min} + c_{\rm max}}.
$$

The speeds $c_\min$ and $c_\max$ are characteristic speeds that waves can travel through the plasma. In GRFFE, the expressions defining them reduce a function of only the metric quantities. $c_\min$ is the negative of the minimum amongst the speeds $c_-$ and $0$ and $c_\max$ is the maximum amongst the speeds $c_+$ and $0$. The speeds $c_\pm = \sqrt{b^2-4ac}$ must be calculated on both the left and right faces, where
$$a = 1/\alpha^2,$$
$$b = 2 \beta^i / \alpha^2$$
and $$c = g^{ii} - (\beta^i)^2/\alpha^2.$$
An outline of a general finite-volume method is as follows, with the current step in bold:
1. The Reconstruction Step - Piecewise Parabolic Method
    1. Within each cell, fit to a function that conserves the volume in that cell using information from the neighboring cells
        * For PPM, we will naturally use parabolas
    1. Use that fit to define the state at the left and right interface of each cell
    1. Apply a slope limiter to mitigate Gibbs phenomenon
1. Interpolate the value of the metric gridfunctions on the cell faces
1. **Solving the Riemann Problem - Harten, Lax, (This notebook, $E_i$ only)**
    1. **Use the left and right reconstructed states to calculate the unique state at boundary**

We will assume in this notebook that the reconstructed velocities and magnetic fields are available on cell faces as input. We will also assume that the metric gridfunctions have been interpolated on the metric faces.

Solving the Riemann problem, then, consists of two substeps: First, we compute the flux through each face of the cell. Then, we add the average of these fluxes to the right-hand side of the evolution equation for the vector potential.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#prelim): Preliminaries
1. [Step 2](#a_i_flux): Computing the Magnetic Flux
    1. [Step 2.a](#hydro_speed): GRFFE characteristic wave speeds
    1. [Step 2.b](#fluxes): Compute the HLLE fluxes
1. [Step 3](#code_validation): Code Validation against `GiRaFFE_NRPy.Afield_flux` NRPy+ Module
1. [Step 4](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Preliminaries \[Back to [top](#toc)\]
$$\label{prelim}$$

We begin by importing the NRPy+ core functionality. We also import the Levi-Civita symbol, the GRHD module, and the GRFFE module.

```python
# Step 0: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

from outputC import *            # NRPy+: Core C code output module
import finite_difference as fin  # NRPy+: Finite difference C code generation module
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import grid as gri               # NRPy+: Functions having to do with numerical grids
import loop as lp                # NRPy+: Generate C code loops
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm   # NRPy+: Reference metric support
import cmdline_helper as cmd     # NRPy+: Multi-platform Python command-line interface
import shutil, os, sys           # Standard Python modules for multiplatform OS-level functions

thismodule = "GiRaFFE_NRPy-Afield_flux"

import GRHD.equations as GRHD
import GRFFE.equations as GRFFE
```

# Step 2: Computing the Magnetic Flux \[Back to [top](#toc)\]
$$\label{a_i_flux}$$

## Step 2.a: GRFFE characteristic wave speeds \[Back to [top](#toc)\]
$$\label{hydro_speed}$$

This function is identical to the one written for Stilde_flux. See [that tutorial](Tutorial-GiRaFFE_NRPy-Stilde-flux.ipynb#hydro_speed) for further information on the derivation.

```python
# We'll write this as a function so that we can calculate the expressions on-demand for any choice of i
def find_cp_cm(lapse,shifti,gammaUUii):
    # Inputs:  u0,vi,lapse,shift,gammadet,gupii
    # Outputs: cplus,cminus

    # a = 1/(alpha^2)
    a = sp.sympify(1)/(lapse*lapse)
    # b = 2 beta^i / alpha^2
    b = sp.sympify(2) * shifti /(lapse*lapse)
    # c = -g^{ii} + (beta^i)^2 / alpha^2
    c = - gammaUUii + shifti*shifti/(lapse*lapse)

    # Now, we are free to solve the quadratic equation as usual. We take care to avoid passing a
    # negative value to the sqrt function.
    detm = b*b - sp.sympify(4)*a*c

    import Min_Max_and_Piecewise_Expressions as noif
    detm = sp.sqrt(noif.max_noif(sp.sympify(0),detm))
    global cplus,cminus
    cplus  = sp.Rational(1,2)*(-b/a + detm/a)
    cminus = sp.Rational(1,2)*(-b/a - detm/a)
```

This function is identical to the one written for Stilde_flux. For more information, see [here](Tutorial-GiRaFFE_NRPy-Stilde-flux.ipynb#fluxes).

```python
# We'll write this as a function, and call it within HLLE_solver, below.
def find_cmax_cmin(field_comp,gamma_faceDD,beta_faceU,alpha_face):
    # Inputs:  flux direction field_comp, Inverse metric gamma_faceUU, shift beta_faceU,
    #          lapse alpha_face, metric determinant gammadet_face
    # Outputs: maximum and minimum characteristic speeds cmax and cmin
    # First, we need to find the characteristic speeds on each face
    gamma_faceUU,unusedgammaDET = ixp.generic_matrix_inverter3x3(gamma_faceDD)
    find_cp_cm(alpha_face,beta_faceU[field_comp],gamma_faceUU[field_comp][field_comp])
    cpr = cplus
    cmr = cminus
    find_cp_cm(alpha_face,beta_faceU[field_comp],gamma_faceUU[field_comp][field_comp])
    cpl = cplus
    cml = cminus

    # The following algorithms have been verified with random floats:

    global cmax,cmin
    # Now, we need to set cmax to the larger of cpr,cpl, and 0

    import Min_Max_and_Piecewise_Expressions as noif
    cmax = noif.max_noif(noif.max_noif(cpr,cpl),sp.sympify(0))

    # And then, set cmin to the smaller of cmr,cml, and 0
    cmin = -noif.min_noif(noif.min_noif(cmr,cml),sp.sympify(0))
```

## Step 2.b: Compute the HLLE fluxes \[Back to [top](#toc)\]
$$\label{fluxes}$$

Here, we we calculate the flux and state vectors for the electric field. The flux vector is here given as
$$
F^i(B^j) = \sqrt{\gamma} (v^i B^j - v^j B^i).
$$
Here, $v^i$ is the drift velocity and $B^i$ is the magnetic field.
This can be easily handled for an input flux direction $i$ with
$$
F^i(B^j) = \epsilon_{ijk} v^j B^k,
$$
where $\epsilon_{ijk} = \sqrt{\gamma} [ijk]$ and $[ijk]$ is the Levi-Civita symbol.

The state vector is simply the magnetic field $B^j$.

```python
def calculate_flux_and_state_for_Induction(field_comp,flux_dirn, gammaDD,betaU,alpha,ValenciavU,BU):
    # Define Levi-Civita symbol
    def define_LeviCivitaSymbol_rank3(DIM=-1):
        if DIM == -1:
            DIM = par.parval_from_str("DIM")

        LeviCivitaSymbol = ixp.zerorank3()

        for i in range(DIM):
            for j in range(DIM):
                for k in range(DIM):
                    # From https://codegolf.stackexchange.com/questions/160359/levi-civita-symbol :
                    LeviCivitaSymbol[i][j][k] = (i - j) * (j - k) * (k - i) * sp.Rational(1,2)
        return LeviCivitaSymbol
    GRHD.compute_sqrtgammaDET(gammaDD)
    # Here, we import the Levi-Civita tensor and compute the tensor with lower indices
    LeviCivitaDDD = define_LeviCivitaSymbol_rank3()
    for i in range(3):
        for j in range(3):
            for k in range(3):
                LeviCivitaDDD[i][j][k] *= GRHD.sqrtgammaDET

    global U,F
    # Flux F = \epsilon_{ijk} v^j B^k
    F = sp.sympify(0)
    for j in range(3):
        for k in range(3):
            F += LeviCivitaDDD[field_comp][j][k] * (alpha*ValenciavU[j]-betaU[j]) * BU[k]
    # U = B^i
    U = BU[flux_dirn]
```

Now, we write a standard HLLE solver based on eq. 3.15 in [the HLLE paper](https://epubs.siam.org/doi/pdf/10.1137/1025002),
$$
F^{\rm HLL} = \frac{c_{\rm min} F_{\rm R} + c_{\rm max} F_{\rm L} - c_{\rm min} c_{\rm max} (U_{\rm R}-U_{\rm L})}{c_{\rm min} + c_{\rm max}}
$$

```python
def HLLE_solver(cmax, cmin, Fr, Fl, Ur, Ul):
    # This solves the Riemann problem for the flux of E_i in one direction

    # F^HLL = (c_\min f_R + c_\max f_L - c_\min c_\max ( st_j_r - st_j_l )) / (c_\min + c_\max)
    return (cmin*Fr + cmax*Fl - cmin*cmax*(Ur-Ul) )/(cmax + cmin)
```

Here, we will use the function we just wrote to calculate the flux through a face. We will pass the reconstructed Valencia 3-velocity and magnetic field on either side of an interface to this function (designated as the "left" and "right" sides) along with the value of the 3-metric, shift vector, and lapse function on the interface. The parameter `flux_dirn` specifies which face through which we are calculating the flux. However, unlike when we used this method to calculate the flux term, the RHS of each component of $A_i$ does not depend on all three of the flux directions. Instead, the flux of one component of the $E_i$ field depends on flux through the faces in the other two directions. This will be handled when we generate the C function, as demonstrated in the example code after this next function.

Note that we allow the user to declare their own gridfunctions if they wish, and default to declaring basic symbols if they are not provided. The default names are chosen to imply interpolation of the metric gridfunctions and reconstruction of the primitives.

```python
def calculate_E_i_flux(flux_dirn,inputs_provided=True,alpha_face=None,gamma_faceDD=None,beta_faceU=None,\
                       Valenciav_rU=None,B_rU=None,Valenciav_lU=None,B_lU=None):
    if not inputs_provided:
        # declare all variables
        alpha_face = sp.symbols(alpha_face)
        beta_faceU = ixp.declarerank1("beta_faceU")
        gamma_faceDD = ixp.declarerank2("gamma_faceDD","sym01")
        Valenciav_rU = ixp.declarerank1("Valenciav_rU")
        B_rU = ixp.declarerank1("B_rU")
        Valenciav_lU = ixp.declarerank1("Valenciav_lU")
        B_lU = ixp.declarerank1("B_lU")
    global E_fluxD
    E_fluxD = ixp.zerorank1()
    for field_comp in range(3):
        find_cmax_cmin(field_comp,gamma_faceDD,beta_faceU,alpha_face)
        calculate_flux_and_state_for_Induction(field_comp,flux_dirn, gamma_faceDD,beta_faceU,alpha_face,\
                                               Valenciav_rU,B_rU)
        Fr = F
        Ur = U
        calculate_flux_and_state_for_Induction(field_comp,flux_dirn, gamma_faceDD,beta_faceU,alpha_face,\
                                               Valenciav_lU,B_lU)
        Fl = F
        Ul = U
        E_fluxD[field_comp] += HLLE_solver(cmax, cmin, Fr, Fl, Ur, Ul)
```

Below, we will write some example code to use the above functions to generate C code for `GiRaFFE_NRPy`. We need to write our own memory reads and writes because we need to add contributions from *both* faces in a given direction, which is expressed in the code as adding contributions from adjacent gridpoints to the RHS, which is not something `FD_outputC` can handle. The `.replace()` function calls adapt these reads and writes to the different directions. Note that, for reconstructions in a given direction, the fluxes are only added to the other two components, as can be seen in the equations we are implementing.
\begin{align}
-E_x(x_i,y_j,z_k) &= \frac{1}{4} \left( F_{\rm HLL}^y(B^z)_{(i,j+1/2,k)}+F_{\rm HLL}^y(B^z)_{(i,j-1/2,k)}-F_{\rm HLL}^z(B^y)_{(i,j,k+1/2)}-F_{\rm HLL}^z(B^y)_{(i,j,k-1/2)} \right) \\
-E_y(x_i,y_j,z_k) &= \frac{1}{4} \left( -F_{\rm HLL}^x(B^z)_{(i+1/2,j,k)}-F_{\rm HLL}^x(B^z)_{(i-1/2,j,k)}+F_{\rm HLL}^z(B^x)_{(i,j,k+1/2)}+F_{\rm HLL}^z(B^x)_{(i,j,k-1/2)} \right) \\
-E_x(x_i,y_j,z_k) &= \frac{1}{4} \left( F_{\rm HLL}^x(B^y)_{(i+1/2,j,k)}+F_{\rm HLL}^x(B^y)_{(i-1/2,j,k)}-F_{\rm HLL}^y(B^x)_{(i,j+1/2,k)}-F_{\rm HLL}^y(B^x)_{(i,j-1/2,k)} \right). \\
\end{align}
From this, we can see that when, for instance, we reconstruct and interpolate in the $x$-direction, we must add only to the $y$- and $z$-components of the electric field.

```python
Memory_Read = """const double alpha_face = auxevol_gfs[IDX4S(ALPHA_FACEGF, i0,i1,i2)];
const double gamma_faceDD00 = auxevol_gfs[IDX4S(GAMMA_FACEDD00GF, i0,i1,i2)];
const double gamma_faceDD01 = auxevol_gfs[IDX4S(GAMMA_FACEDD01GF, i0,i1,i2)];
const double gamma_faceDD02 = auxevol_gfs[IDX4S(GAMMA_FACEDD02GF, i0,i1,i2)];
const double gamma_faceDD11 = auxevol_gfs[IDX4S(GAMMA_FACEDD11GF, i0,i1,i2)];
const double gamma_faceDD12 = auxevol_gfs[IDX4S(GAMMA_FACEDD12GF, i0,i1,i2)];
const double gamma_faceDD22 = auxevol_gfs[IDX4S(GAMMA_FACEDD22GF, i0,i1,i2)];
const double beta_faceU0 = auxevol_gfs[IDX4S(BETA_FACEU0GF, i0,i1,i2)];
const double beta_faceU1 = auxevol_gfs[IDX4S(BETA_FACEU1GF, i0,i1,i2)];
const double beta_faceU2 = auxevol_gfs[IDX4S(BETA_FACEU2GF, i0,i1,i2)];
const double Valenciav_rU0 = auxevol_gfs[IDX4S(VALENCIAV_RU0GF, i0,i1,i2)];
const double Valenciav_rU1 = auxevol_gfs[IDX4S(VALENCIAV_RU1GF, i0,i1,i2)];
const double Valenciav_rU2 = auxevol_gfs[IDX4S(VALENCIAV_RU2GF, i0,i1,i2)];
const double B_rU0 = auxevol_gfs[IDX4S(B_RU0GF, i0,i1,i2)];
const double B_rU1 = auxevol_gfs[IDX4S(B_RU1GF, i0,i1,i2)];
const double B_rU2 = auxevol_gfs[IDX4S(B_RU2GF, i0,i1,i2)];
const double Valenciav_lU0 = auxevol_gfs[IDX4S(VALENCIAV_LU0GF, i0,i1,i2)];
const double Valenciav_lU1 = auxevol_gfs[IDX4S(VALENCIAV_LU1GF, i0,i1,i2)];
const double Valenciav_lU2 = auxevol_gfs[IDX4S(VALENCIAV_LU2GF, i0,i1,i2)];
const double B_lU0 = auxevol_gfs[IDX4S(B_LU0GF, i0,i1,i2)];
const double B_lU1 = auxevol_gfs[IDX4S(B_LU1GF, i0,i1,i2)];
const double B_lU2 = auxevol_gfs[IDX4S(B_LU2GF, i0,i1,i2)];
REAL A_rhsD0 = 0; REAL A_rhsD1 = 0; REAL A_rhsD2 = 0;
"""
Memory_Write = """rhs_gfs[IDX4S(AD0GF,i0,i1,i2)] += A_rhsD0;
rhs_gfs[IDX4S(AD1GF,i0,i1,i2)] += A_rhsD1;
rhs_gfs[IDX4S(AD2GF,i0,i1,i2)] += A_rhsD2;
"""

indices = ["i0","i1","i2"]
indicesp1 = ["i0+1","i1+1","i2+1"]

subdir = "RHSs"
for flux_dirn in range(3):
    Af.calculate_E_i_flux(flux_dirn,True,alpha_face,gamma_faceDD,beta_faceU,\
                          Valenciav_rU,B_rU,Valenciav_lU,B_lU)

    E_field_to_print = [\
                        sp.Rational(1,4)*Af.E_fluxD[(flux_dirn+1)%3],\
                        sp.Rational(1,4)*Af.E_fluxD[(flux_dirn+2)%3],\
                       ]
    E_field_names = [\
                     "A_rhsD"+str((flux_dirn+1)%3),\
                     "A_rhsD"+str((flux_dirn+2)%3),\
                    ]

    desc = "Calculate the electric flux on the left face in direction " + str(flux_dirn) + "."
    name = "calculate_E_field_D" + str(flux_dirn) + "_right"
    outCfunction(
        outfile  = os.path.join(out_dir,subdir,name+".h"), desc=desc, name=name,
        params   ="const paramstruct *params,const REAL *auxevol_gfs,REAL *rhs_gfs",
        body     =  Memory_Read \
                   +outputC(E_field_to_print,E_field_names,"returnstring",params="outCverbose=False").replace("IDX4","IDX4S")\
                   +Memory_Write,
        loopopts ="InteriorPoints",
        rel_path_for_Cparams=os.path.join("../"))

    desc = "Calculate the electric flux on the left face in direction " + str(flux_dirn) + "."
    name = "calculate_E_field_D" + str(flux_dirn) + "_left"
    outCfunction(
        outfile  = os.path.join(out_dir,subdir,name+".h"), desc=desc, name=name,
        params   ="const paramstruct *params,const REAL *auxevol_gfs,REAL *rhs_gfs",
        body     =  Memory_Read.replace(indices[flux_dirn],indicesp1[flux_dirn]) \
                   +outputC(E_field_to_print,E_field_names,"returnstring",params="outCverbose=False").replace("IDX4","IDX4S")\
                   +Memory_Write,
        loopopts ="InteriorPoints",
        rel_path_for_Cparams=os.path.join("../"))
```

# Step 3:  Code Validation against `GiRaFFE_NRPy.Induction_Equation` NRPy+ Module \[Back to [top](#toc)\]
$$\label{code_validation}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for the $\texttt{GiRaFFE}$ evolution equations and auxiliary quantities we intend to use between
1. this tutorial and
2. the NRPy+ [GiRaFFE_NRPy.Induction_Equation](../../edit/in_progress/GiRaFFE_NRPy/Induction_Equation.py) module.

Below are the gridfunction registrations we will need for testing. We will pass these to the above functions to self-validate the module that corresponds with this tutorial.

```python
all_passed=True
def comp_func(expr1,expr2,basename,prefixname2="C2P_P2C."):
    if str(expr1-expr2)!="0":
        print(basename+" - "+prefixname2+basename+" = "+ str(expr1-expr2))
        all_passed=False

def gfnm(basename,idx1,idx2=None,idx3=None):
    if idx2==None:
        return basename+"["+str(idx1)+"]"
    if idx3==None:
        return basename+"["+str(idx1)+"]["+str(idx2)+"]"
    return basename+"["+str(idx1)+"]["+str(idx2)+"]["+str(idx3)+"]"

# These are the standard gridfunctions we've used before.
#ValenciavU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","ValenciavU",DIM=3)
#gammaDD = ixp.register_gridfunctions_for_single_rank2("AUXEVOL","gammaDD","sym01")
#betaU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","betaU")
#alpha = gri.register_gridfunctions("AUXEVOL",["alpha"])
#AD = ixp.register_gridfunctions_for_single_rank1("EVOL","AD",DIM=3)
#BU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","BU",DIM=3)

# We will pass values of the gridfunction on the cell faces into the function. This requires us
# to declare them as C parameters in NRPy+. We will denote this with the _face infix/suffix.
alpha_face = gri.register_gridfunctions("AUXEVOL","alpha_face")
gamma_faceDD = ixp.register_gridfunctions_for_single_rank2("AUXEVOL","gamma_faceDD","sym01")
beta_faceU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","beta_faceU")

# We'll need some more gridfunctions, now, to represent the reconstructions of BU and ValenciavU
# on the right and left faces
Valenciav_rU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","Valenciav_rU",DIM=3)
B_rU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","B_rU",DIM=3)
Valenciav_lU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","Valenciav_lU",DIM=3)
B_lU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","B_lU",DIM=3)

import GiRaFFE_NRPy.Afield_flux as Af

expr_list = []
exprcheck_list = []
namecheck_list = []

for flux_dirn in range(3):
    calculate_E_i_flux(flux_dirn,True,alpha_face,gamma_faceDD,beta_faceU,\
                       Valenciav_rU,B_rU,Valenciav_lU,B_lU)
    Af.calculate_E_i_flux(flux_dirn,True,alpha_face,gamma_faceDD,beta_faceU,\
                          Valenciav_rU,B_rU,Valenciav_lU,B_lU)
    namecheck_list.extend([gfnm("E_fluxD",flux_dirn)])
    exprcheck_list.extend([Af.E_fluxD[flux_dirn]])
    expr_list.extend([E_fluxD[flux_dirn]])

for mom_comp in range(len(expr_list)):
    comp_func(expr_list[mom_comp],exprcheck_list[mom_comp],namecheck_list[mom_comp])

import sys
if all_passed:
    print("ALL TESTS PASSED!")
else:
    print("ERROR: AT LEAST ONE TEST DID NOT PASS")
    sys.exit(1)
```

    ALL TESTS PASSED!

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-GiRaFFE_NRPy-Induction_Equation.pdf](Tutorial-GiRaFFE_NRPy-Induction_Equation.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx --log-level='WARN' Tutorial-GiRaFFE_NRPy-Afield_flux.ipynb
!pdflatex -interaction=batchmode Tutorial-GiRaFFE_NRPy-Afield_flux.tex
!pdflatex -interaction=batchmode Tutorial-GiRaFFE_NRPy-Afield_flux.tex
!pdflatex -interaction=batchmode Tutorial-GiRaFFE_NRPy-Afield_flux.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


# Session 1: An introduction to Python and Jupyter Notebooks

------------------------------------------------------
*Introduction to Data Science & Machine Learning*

*Pablo M. Olmos olmos@tsc.uc3m.es*

------------------------------------------------------

# Python 101

In this notebook we will review basic things you need to know if you want to work with Python as you do with Matlab. We will skip tons of things, we'll learn many more things on the fly.

First of all, in Python notebooks you have text boxes "Markdonws" where you can put text, images, and even latex equations (**double click** here to see the actual [markdown code](http://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Working%20With%20Markdown%20Cells.html)).

\begin{align}
E=mc^2
\end{align}

You can execute each code box by typing "Shift+Enter"

## Let's first take a look on Python data types, and how to print stuff on the screen

This is only a quick summary. For a full description, take a look to the [Python built-in types](Built-in Types).

### Standard integer and float variables

```python
## Standard Python data formats

a=5 #This is an integer variable

b=8.434 #This is a float variable

print ('"a" is an %r variable, "b" is a %r variable' %(type(a),type(b)))
```

    "a" is an <class 'int'> variable, "b" is a <class 'float'> variable

### Creating string variables

* Strings in Python can be simple or double quotes
* Concatenating strings

```python
s="Hello " + "world"
print(s)
```

    Hello world

```python
s='Hello ' + 'world'
print(s)
```

    Hello world

```python
s='spam'
len(s)
```

    4

```python
print(s[0],s[-1])
```

    s m

### Lists

[Lists](https://docs.python.org/3.6/library/stdtypes.html?highlight=lists#lists) are what they seem - a list of values. Each one of them is numbered, starting from zero. Elements in a list can contain any data type. This is how you can create and index a list:

```python
my_list=[3, 5, 6, 'string', [2,4,6]]

# Adding a new element (another list containing both a and b defined above):

print (" The first element is %r" %(my_list[0]))

print (" The fourth element is =%r" %(my_list[3]))

print (" The fith element is actually another list %r \n" %(my_list[4]))

```

     The first element is 3
     The fourth element is ='string'
     The fith element is actually another list [2, 4, 6]

Lists object provide many different methods to manage them, see [here](https://docs.python.org/3.6/tutorial/datastructures.html#) for a full documentation:

```python
my_list.append([a,b])

print(my_list.pop(2))
```

    6

## Dictionaries

Unlike sequences, which are indexed by a range of numbers, [dictionaries](https://docs.python.org/2.7/tutorial/datastructures.html#dictionaries) are indexed by *keys*, which can be any immutable type; strings and numbers can always be keys.

It is best to think of a dictionary as an unordered set of key: value pairs, with the requirement that the keys are unique (within one dictionary). A pair of braces creates an empty dictionary: `{}`.

```python
my_dictionary={'key 1': 6, 'key 2': my_list}

l = list(my_dictionary.keys()) # List of keys

print ("The first key is %r, and the second key is %r\n" %(l[0],l[1]))

print ("The value associated with %r is %r\n" %(l[0],my_dictionary['key 2']))

```

    The first key is 'key 1', and the second key is 'key 2'

    The value associated with 'key 1' is [3, 5, 'string', [2, 4, 6], [5, 8.434]]

```python
# Another example, a phone book

d = {'Antonio':8806336, \
'Eva':6784346, 'Alex':7658344, \
'Pedro':1122345}
```

```python
# Adding new elements
d['Alicia']=2365872
d
```

    {'Alex': 7658344,
     'Alicia': 2365872,
     'Antonio': 8806336,
     'Eva': 6784346,
     'Pedro': 1122345}

```python
# Deleting elements
del d['Eva']
d
```

    {'Alex': 7658344, 'Alicia': 2365872, 'Antonio': 8806336, 'Pedro': 1122345}

## Loops and function definitions

Be careful with indentation!!!

The following code generates a function that takes three inputs (a,b,c). For any integer i in the set [a,b], determines if it is divisible by c.

```python
def my_function(a,b,c):
    for i in range(a,b):

        if(i % c==0):
            print ("a=%r is divisible by c=%r, and belongs to the set [%r,%r] \n" %(i,c,a,b))

    print ("This is not part of the foor loop\n")

```

```python
## Now we run the function defined above
my_function(10,22,3)
```

    a=12 is divisible by c=3, and belongs to the set [10,22]

    a=15 is divisible by c=3, and belongs to the set [10,22]

    a=18 is divisible by c=3, and belongs to the set [10,22]

    a=21 is divisible by c=3, and belongs to the set [10,22]

    This is not part of the foor loop

Lists and dictionaries are very easy to loop through, inspect what the following code does:

```python
contador = 0
for item in my_list:
    contador += 1
    print(item, '\n')

for i,item in enumerate(my_list):
    print('The %d-th entry in the list is %r\n' %(i,item))
```

    3

    5

    string

    [2, 4, 6]

    [5, 8.434]

    The 0-th entry in the list is 3

    The 1-th entry in the list is 5

    The 2-th entry in the list is 'string'

    The 3-th entry in the list is [2, 4, 6]

    The 4-th entry in the list is [5, 8.434]

```python
my_list[3][1]
```

    4

To loop over two or more sequences at the same time, the entries can be paired with the `zip()` function.

```python
values = [20.5,30,8000,100]

items = ['lamp', 'table','Fully-equipped kitchen','Fancy Mirror']

for v, i in zip(values, items):
    print('The price of the item %r in the store is %d €.\n' %(i, v))
```

    The price of the item 'lamp' in the store is 20 €.

    The price of the item 'table' in the store is 30 €.

    The price of the item 'Fully-equipped kitchen' in the store is 8000 €.

    The price of the item 'Fancy Mirror' in the store is 100 €.

In Python 3.X, looping through a dictionary is as follows:

```python
for key, value in d.items():
    print('The phone number of %s is %d\n' %(key,value))
```

    The phone number of Antonio is 8806336

    The phone number of Alex is 7658344

    The phone number of Pedro is 1122345

    The phone number of Alicia is 2365872

## Conditional Loops

```python
count = 0
while (count < 9):
    print('The count is:' , count)
    count = count + 1

print("Good bye!")
```

    The count is: 0
    The count is: 1
    The count is: 2
    The count is: 3
    The count is: 4
    The count is: 5
    The count is: 6
    The count is: 7
    The count is: 8
    Good bye!

```python
for letter in 'Python':
    if(letter != 'o'):
        print('Current Letter :', letter)
```

    Current Letter : P
    Current Letter : y
    Current Letter : t
    Current Letter : h
    Current Letter : n