```julia
using Pkg
Pkg.activate("presentation_ebs")
Pkg.add(["BenchmarkTools", "MATLAB", "LoopVectorization",
         "MarketData", "TimeSeries", "Distributions",
         "KernelDensity", "StatsPlots", "ProgressMeter",
         "PyCall", "Conda", "ARCHModels"])
using Conda
Conda.add("arch-py", channel="conda-forge")
Pkg.update()

```

      Activating environment at `C:\Users\sabro\Documents\julia\notebooks\presentation\presentation_ebs\Project.toml`
        Updating registry at `C:\Users\sabro\.julia\registries\General`
        Updating git-repo `https://github.com/JuliaRegistries/General.git`
       Resolving package versions...
      No Changes to `C:\Users\sabro\Documents\julia\notebooks\presentation\presentation_ebs\Project.toml`
      No Changes to `C:\Users\sabro\Documents\julia\notebooks\presentation\presentation_ebs\Manifest.toml`
    ┌ Info: Running `conda install -y -c conda-forge arch-py` in root environment
    └ @ Conda C:\Users\sabro\.julia\packages\Conda\sNGum\src\Conda.jl:128

    Collecting package metadata (current_repodata.json): ...working... done
    Solving environment: ...working... done

    # All requested packages already installed.

        Updating registry at `C:\Users\sabro\.julia\registries\General`
        Updating git-repo `https://github.com/JuliaRegistries/General.git`
      No Changes to `C:\Users\sabro\Documents\julia\notebooks\presentation\presentation_ebs\Project.toml`
      No Changes to `C:\Users\sabro\Documents\julia\notebooks\presentation\presentation_ebs\Manifest.toml`

#####
 The Unreasonable Effectiveness of Julia

 Simon A. Broda

 University of Amsterdam and Hochschule Luzern

s.a.broda@uva.nl

 This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No. 750559).

# Outline
* The Julia Language
* Refresher on ARCH Models
* The `ARCHModels.jl` Package
   * Usage
   * Benchmarks

These slides are available at https://github.com/s-broda/presentation.

# The Julia Language
## General Information
* New programming language started at MIT.
* Designed with scientific computing in mind.
* Version 1.0 released in August 2018 after 9 years of development. Current version is 1.6.
* Free and open source software. Available at https://julialang.org/.
* In the words of its creators,
> We want a language that’s open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a
language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want
something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as
powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn,
yet keeps the most serious hackers happy. We want it interactive and we want it compiled.

## Highlights for Scientists
* Interactive REPL (like Matlab, Python, R, etc.) allows for exploratory analysis, rapid prototyping.
* Unlike these, Julia is JIT compiled, hence fast (typically within a few percent of C)
* Syntax superficially similar to Matlab.
* Rich type system, multiple dispatch.
* Fast-growing eco-system with many state-of-the-art packages (e.g., `ForwardDiff.jl`, `DifferentialEquations.jl`, `JuMP`, ...).

## A Small Taste of Julia

* Let's say we want to implement a function that sums an array:

```julia
function mysum(x)
    s = zero(eltype(x))
    for i in eachindex(x)
        s += x[i]
    end
    s
end
mysum([1., 2., 3.])
```

    6.0

* Let's benchmark it:

```julia
x = randn(10^7);
```

```julia
using BenchmarkTools
@btime mysum(x); # @btime runs the code a large number of times
```

      7.458 ms (1 allocation: 16 bytes)

* For comparison, the built-in function:

```julia
@btime sum(x);
```

      2.680 ms (1 allocation: 16 bytes)

* Close, but we are 2-3x slower.
* But we can do better! 2nd attempt:

```julia
function mysum(x)
    s = zero(eltype(x))
    @inbounds @simd for i in eachindex(x) # @simd exploits 'single instruction multiple data', a type of parallelism
        s += x[i]
    end
    s
end
mysum([1., 2., 3.])
```

    6.0

```julia
@btime mysum(x);
```

      2.638 ms (1 allocation: 16 bytes)

* Not bad: by just adding a simple decorator, we now match the speed of the built-in function!
* This is, in fact, expected: the built-in function is implemented in Julia, like most of the standard library.
* We can even look at the code:

```julia
@which sum(x)
```

sum(a::AbstractArray; dims, kw...) in Base at reducedim.jl:873

## One more thing...

```julia
using LoopVectorization
function mysum(x)
    s = zero(eltype(x))
    @tturbo for i in eachindex(x)
        s += x[i]
    end
    s
end
mysum([1., 2., 3.])
```

    6.0

```julia
@btime mysum(x);
```

      948.600 μs (1 allocation: 16 bytes)

* __Wow :o__
* This difference comes from multi-threading; @tturbo exploits all the cores the machine has available.

# Refresher on ARCH Models
* Daily financial returns data exhibit a number of *stylized facts*:
  * Volatility clustering
  * Non-Gaussianity, fat tails
  * Leverage effects: negative returns increase future volatility
* Other types of data (e.g., changes in interest rates) exhibit similar phenomena.
* These effects are important in many areas in finance, in particular in risk management.
* [G]ARCH ([**G**eneralized] **A**utoregressive **C**onditional **V**olatility) models are the most popular for modelling them.

## Example: volatility clustering in AAPL returns
* The `MarketData` package allows us to download historical data from Yahoo Finance.

```julia
using MarketData, TimeSeries
r = percentchange(yahoo(:AAPL)[:AdjClose]) # returns a TimeArray
data = values(r) # an array containing just the plain data

if !isfile("returns.svg") || !isfile("kde.svg")
    using Distributions, KernelDensity, StatsPlots
    plot(r, title="Volatility Clustering", legend=:none, ylabel="\$r_t\$")
    savefig("returns.svg")
    plot(kde(data), label="Kernel Density", title="Fat Tails")
    plot!(fit(Normal, data), label="Fitted Normal")
    savefig("kde.svg")
end
```

## (G)ARCH Models
* Basic setup: given a sample of financial returns $\{r_t\}_{t\in\{1,\ldots,T\}}$, decompose $r_t$ as
$$
r_t=\mu_t+\sigma_tz_t, \quad z_t\stackrel{i.i.d.}{\sim}(0,1),
$$
where $\mu_t\equiv\mathbb{E}[r_t\mid \mathcal{F}_{t-1}]$ and $\sigma_t^2\equiv \mathbb{E}[(r_t-\mu_t)^2\mid \mathcal{F}_{t-1}]$.
* Assume $\mu_t=0$ for simplicity. Focus is on the *volatility* $\sigma_t$. G(ARCH) models make $\sigma_t$ a function of *past* returns and variances. Examples:

## Examples
* ARCH(q) (Engle, Econometrica 1982):
$$\sigma_t^2=\omega+\sum_{i=1}^q \alpha_ir_{t-i}^2%,\quad \omega,\alpha_i>0,\quad \sum_{i=1}^q\alpha_i<1.
$$
* GARCH(p, q) (Bollerslev, JoE 1986)
$$\sigma_t^2=\omega+ \sum_{i=1}^p\beta_{i}\sigma_{t-i}^2 + \sum_{i=1}^q\alpha_ir_{t-i}^2%,\quad \omega,\alpha_i,\beta_i>0,\quad \sum_{i=1}^{\max p,q} \alpha_i+\beta_i<1.
$$
* EGARCH(o, p, q) (Nelson, Econometrica 1991)
$$\log(\sigma_t^2)=\omega+\sum_{i=1}^o\gamma_{i}z_{t-i}+\sum_{i=1}^p\beta_i\log(\sigma_{t-i}^2)+\sum_{i=1}^q \alpha_i (|z_t|-\mathbb{E}|z_t|)%, \quad \sum_{i=1}^p \beta_i<0.
$$
* and many (many, many...) others

## Estimation
* G(ARCH) models are usually estimated by maximum likelihood: with $f_z$ denoting the density of $z_t$,
$$\max \sum_t \log f(r_t\mid \mathcal{F}_{t-1})=\max \sum_t \log f_z(r_t/\sigma_t)-\log \sigma_t.$$
* Recursive nature of $\sigma_t$ means the computation cannot be "vectorized" $\Rightarrow$ loops.
* Julia is very well suited for this. Matlab (and the `rugarch` package for Python) have to implement the likelihood in C.

# The `ARCHModels.jl` Package
## Installation
* `ARCHModels.jl` is a registered julia package. To install it, just do `using Pkg; Pkg.add("ARCHModels")`
*  Extensive documentation available at https://s-broda.github.io/ARCHModels.jl/stable/.

## Key Features
* Supports simulating, estimating, forecasting, and backtesting ARCH models.
* Currently: ARCH, GARCH, TGARCH, and EGARCH models of arbitrary orders, with Gaussian, Student's $t$, and GED errors.
* Regression and ARMA models for the mean equation.
* CCC and DCC models for multivariate returns.
* Entirely written in Julia.
* Designed to be easily extensible with new models and distributions (nice thesis project...)
* Gradients and Hessians (for both numerical maximization of the likelihood and constructing standard errors) are obtained by [automatic differentiation](http://www.autodiff.org/?module=Introduction) via `ForwardDiff.jl`.

## Usage

* The first step in building an ARCH model is usually to test for the presence of volatility clustering.
* `ARCHModels.jl` provides [Engle's (Econometrica 1982)](https://doi.org/10.2307/1912773) ARCH-LM test for this.
* The null is that $\gamma_i=0$ in the auxiliary regression
$$
r_t^2=\alpha+\gamma_1 r_{t-1}^2+\gamma_2 r_{t-2}^2+\cdots+\gamma_p r_{t-p}^2+\epsilon_t.
$$

```julia
using ARCHModels
ARCHLMTest(data, 4) # p = 4 lags
```

    ARCH LM test for conditional heteroskedasticity
    -----------------------------------------------
    Population details:
        parameter of interest:   T⋅R² in auxiliary regression
        value under h_0:         0
        point estimate:          147.398

    Test summary:
        outcome with 95% confidence: reject h_0
        p-value:                     <1e-30

    Details:
        sample size:                    10205
        number of lags:                 4
        LM statistic:                   147.398

* Unsurprisingly, the test rejects.

* Fitting a GARCH model is as simple as

```julia
model = fit(GARCH{1, 1}, data)
```

    GARCH{1, 1} model with Gaussian errors, T=10205.

    Mean equation parameters:
    ─────────────────────────────────────────────
         Estimate    Std.Error  z value  Pr(>|z|)
    ─────────────────────────────────────────────
    μ  0.00184929  0.000257205  7.18994    <1e-12
    ─────────────────────────────────────────────

    Volatility parameters:
    ─────────────────────────────────────────────
          Estimate  Std.Error   z value  Pr(>|z|)
    ─────────────────────────────────────────────
    ω   5.67925e-6  2.9651e-6   1.91537    0.0554
    β₁  0.929851    0.0136741  68.0008     <1e-99
    α₁  0.0674646   0.0176432   3.82384    0.0001
    ─────────────────────────────────────────────

* Alternatively, the (multithreaded!) `selectmodel` method does automatic model selection (i.e., chooses $p$ and $q$ by minimizing the AIC/AICC/BICC).

```julia
model2 = selectmodel(GARCH, data; maxlags=3, criterion=bic) # optional keyword arguments
```

    GARCH{3, 1} model with Gaussian errors, T=10205.

    Mean equation parameters:
    ─────────────────────────────────────────────
         Estimate    Std.Error  z value  Pr(>|z|)
    ─────────────────────────────────────────────
    μ  0.00201148  0.000241287  8.33647    <1e-16
    ─────────────────────────────────────────────

    Volatility parameters:
    ──────────────────────────────────────────────────
           Estimate   Std.Error      z value  Pr(>|z|)
    ──────────────────────────────────────────────────
    ω   1.34075e-5   1.42247e-6  9.42551        <1e-20
    β₁  0.520858     0.088458    5.88819        <1e-08
    β₂  1.36846e-48  3.42138e-7  3.99972e-42    1.0000
    β₃  0.358345     0.0886664   4.0415         <1e-04
    α₁  0.101772     0.0134813   7.54909        <1e-13
    ──────────────────────────────────────────────────

* The return value is of type `UnivariateARCHModel`:

```julia
typeof(model)
```

    UnivariateARCHModel{Float64, GARCH{1, 1, Float64}, StdNormal{Float64}, Intercept{Float64}}

* `UnivariateARCHModel` supports many useful methods, such as
 `confint`, `aic`, `bic`, `aicc`, `informationmatrix`, `score`, `vcov`:

```julia
aic(model)
```

    -45895.13689151763

* A `UnivariateARCHModel` essentially consists of a volatility specification (of type `UnivariateVolatilitySpec`),
an error distribution (of type `StandardizedDistribution`), and a mean specification (of type `MeanSpec`).
* The following are currently implemented:

```julia
print.(string.(subtypes(UnivariateVolatilitySpec)) .* " ");
```

    EGARCH TGARCH

```julia
print.(string.(subtypes(StandardizedDistribution))[2:end] .* " ");
```

    StdGED StdNormal StdSkewT StdT

```julia
print.(string.(subtypes(MeanSpec)).* " ");
```

    ARMA Intercept NoIntercept Regression

* We can use these to construct an ARCHModel manually:

```julia
T = 10^4
mydata = zeros(T)
volaspec = EGARCH{1, 1, 1}([0.02, .09, .83, .01])
using Random; Random.seed!(1)
mymodel = UnivariateARCHModel(volaspec, mydata; dist=StdT(4.)) # dist is an optional keyword argument
```

    EGARCH{1, 1, 1} model with Student's t errors, T=10000.

    ─────────────────────────────────────────────────
                                  ω    γ₁    β₁    α₁
    ─────────────────────────────────────────────────
    Volatility parameters:     0.02  0.09  0.83  0.01
    ─────────────────────────────────────────────────
    ──────────────────────────────
                                 ν
    ──────────────────────────────
    Distribution parameters:   4.0
    ──────────────────────────────

* With a `UnivariateARCHModel` in hand, we can do things like

```julia
simulate!(mymodel) # by convention, the bang indicates that the method modifies its argument
```

    EGARCH{1, 1, 1} model with Student's t errors, T=10000.

    ─────────────────────────────────────────────────
                                  ω    γ₁    β₁    α₁
    ─────────────────────────────────────────────────
    Volatility parameters:     0.02  0.09  0.83  0.01
    ─────────────────────────────────────────────────
    ──────────────────────────────
                                 ν
    ──────────────────────────────
    Distribution parameters:   4.0
    ──────────────────────────────

```julia
fit!(mymodel)
```

    EGARCH{1, 1, 1} model with Student's t errors, T=10000.

    Volatility parameters:
    ───────────────────────────────────────────────
           Estimate   Std.Error   z value  Pr(>|z|)
    ───────────────────────────────────────────────
    ω    0.0274054   0.00733817   3.73464    0.0002
    γ₁   0.113959    0.0131327    8.67752    <1e-17
    β₁   0.811765    0.0244774   33.1638     <1e-99
    α₁  -0.00637894  0.0153358   -0.41595    0.6774
    ───────────────────────────────────────────────

    Distribution parameters:
    ─────────────────────────────────────────
       Estimate  Std.Error  z value  Pr(>|z|)
    ─────────────────────────────────────────
    ν   3.83768   0.152659   25.139    <1e-99
    ─────────────────────────────────────────

```julia
ARCHLMTest(mymodel) # tests the standardized residuals
```

    ARCH LM test for conditional heteroskedasticity
    -----------------------------------------------
    Population details:
        parameter of interest:   T⋅R² in auxiliary regression
        value under h_0:         0
        point estimate:          0.0873996

    Test summary:
        outcome with 95% confidence: fail to reject h_0
        p-value:                     0.7675

    Details:
        sample size:                    10000
        number of lags:                 1
        LM statistic:                   0.0873996

```julia
predict(mymodel, :volatility) # (:volatility | :variance | :return | :VaR)
```

    0.9582988646322497

## Value at Risk

* In-sample Value-at-Risk estimates are available via the `VaRs` function

```julia
model = fit(GARCH{1, 1}, data)
vars = VaRs(model, 0.05)

if !isfile("VaRplot.svg")
    using Plots
    plot(-data, legend=:none, xlabel="\$t\$", ylabel="\$-r_t\$", title="Value at Risk, In-Sample")
    plot!(vars, color=:purple)
    savefig("VaRplot.svg");
end
```

* How about out-of-sample (backtesting)?
* Requires re-estimating the model at each time step.
* Not built in, but easy to do:

```julia
using ProgressMeter

T = length(data)
windowsize = 1000
vars = similar(data); fill!(vars, NaN)
@showprogress "Fitting $(T-1-windowsize) GARCH models: " for t = windowsize+1:T-1
    m = fit(GARCH{1, 1}, data[t-windowsize:t])
    vars[t+1] = predict(m, :VaR; level=0.05)
end

if !isfile("VaRplot_oos.svg")
    using Plots
    plot(-data, legend=:none, xlabel="\$t\$", ylabel="\$-r_t\$", title="Value at Risk, Out-of-Sample")
    plot!(vars, color=:purple)
    savefig("VaRplot_oos.svg");
end
```

    Fitting 9204 GARCH models: 100%|████████████████████████| Time: 0:00:24

* We can backtest these out-of-sample VaR predictions using Engle and Manganelli's (2004) dynamic quantile test.
* The test is based on a *hit series*
$$I_t\equiv\cases{1, r_t<-VaR_t\\0, \mbox{otherwise.}}$$
* The null is that $\alpha=\beta=\gamma=0$ in the auxiliary regression
$$
I_t-0.05 = \alpha +\beta I_{t-1} + \gamma VaR_t+\epsilon_t.
$$

```julia
DQTest(data[windowsize+1:end], vars[windowsize+1:end], 0.05)
```

    Engle and Manganelli's (2004) DQ test (out of sample)
    -----------------------------------------------------
    Population details:
        parameter of interest:   Wald statistic in auxiliary regression
        value under h_0:         0
        point estimate:          35.834

    Test summary:
        outcome with 95% confidence: reject h_0
        p-value:                     <1e-07

    Details:
        sample size:                    9205
        number of lags:                 1
        VaR level:                      0.05
        DQ statistic:                   35.834

# Benchmarks
  * Bollerslev and Ghysels (JBES 1996) data is de facto standard in comparing implementations of GARCH models.
  * Data consist of daily German mark/British pound exchange rates (1974 observations).
  * Available in `ARCHModels.jl` as the constant `BG96`.

```julia
if !isfile("DMGBP.svg")
    using Plots
    plot(BG96, legend=:none, title="Bollerslev and Ghysels (1996) Data", ylabel="\$r_t\$", xlabel="\$t\$")
    savefig("DMGBP.svg")
end
```

## GARCH
* Fitting in Julia:

```julia
using BenchmarkTools
@btime fit(GARCH{1, 1}, $BG96)
```

      3.208 ms (2546 allocations: 203.33 KiB)

    GARCH{1, 1} model with Gaussian errors, T=1974.

    Mean equation parameters:
    ───────────────────────────────────────────────
          Estimate   Std.Error    z value  Pr(>|z|)
    ───────────────────────────────────────────────
    μ  -0.00616637  0.00920152  -0.670147    0.5028
    ───────────────────────────────────────────────

    Volatility parameters:
    ─────────────────────────────────────────────
         Estimate   Std.Error   z value  Pr(>|z|)
    ─────────────────────────────────────────────
    ω   0.0107606  0.00649303   1.65725    0.0975
    β₁  0.805875   0.0724765   11.1191     <1e-27
    α₁  0.153411   0.0536404    2.86       0.0042
    ─────────────────────────────────────────────

* Now Matlab:

```julia
using MATLAB # this is for calling Matlab from within Julia. I do this because a Jupyter notebook
mat"version" # only supports a single language at a time.
```

    "9.10.0.1669831 (R2021a) Update 2"

```julia
mat"""
estimate(garch('ARCHLags', 1, 'GARCHLags', 1, 'offset', NaN), $BG96); 0;
f = @()estimate(garch('ARCHLags', 1, 'GARCHLags', 1, 'offset', NaN), $BG96, 'Display', 'off');
t = timeit(f);
disp(['Time per run: ', num2str(t * 1000), ' ms'])
"""
```

        GARCH(1,1) Conditional Variance Model with Offset (Gaussian Distribution):

                      Value       StandardError    TStatistic      PValue
                    __________    _____________    __________    __________

        Constant      0.010761       0.001323         8.1342     4.1454e-16
        GARCH{1}       0.80597        0.01656         48.669              0
        ARCH{1}        0.15313       0.013974         10.959     6.0379e-28
        Offset      -0.0061904      0.0084336       -0.73402        0.46294

    Time per run: 33.029 ms

    0.0

* ARCH.jl is faster by a factor of about 10 or more, depending on the machine, despite Matlab calling into compiled C code.
* Estimates are quite similar, but standard errors and $t$-statistics differ.
* So which standard errors are correct? Let's compare with the results from Brooks et. al. (Int. J. Fcst. 2001).
* Brooks et. al. compare implementations of the GARCH(1, 1) model.
* They give the following estimates (**$t$-stats**) as benchmarks:
$\mu=−0.00619$ $(\mathbf{−0.67})$, $\omega=0.0108$ $(\mathbf{1.66})$, $\beta_1=0.806$ $(\mathbf{11.11})$, $\alpha_1=0.153$ $(\mathbf{2.86})$.
* `ARCHModels.jl` is bang on. Not sure what Matlab is doing. Look at those t-stats!

* Finally, Python:

```julia
using PyCall # for calling Python from within Julia
py"""import pandas as pd
import timeit
import functools
from arch.univariate.mean import ConstantMean
from arch.univariate.distribution import Normal
from arch.univariate import GARCH
am = ConstantMean($BG96, rescale=False)
am.volatility = GARCH(1, 0, 1)
N = 100
print(am.fit(disp="off"))
print("Time per run:", timeit.timeit(functools.partial(am.fit, disp="off"), number=N) / N * 1000, "ms") """
```

                         Constant Mean - GARCH Model Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.000
    Mean Model:             Constant Mean   Adj. R-squared:                  0.000
    Vol Model:                      GARCH   Log-Likelihood:               -1104.52
    Distribution:                  Normal   AIC:                           2217.04
    Method:            Maximum Likelihood   BIC:                           2239.39
                                            No. Observations:                 1974
    Date:                Sun, Jun 06 2021   Df Residuals:                     1973
    Time:                        18:42:58   Df Model:                            1
                                      Mean Model
    ==============================================================================
                      coef    std err          t      P>|t|       95.0% Conf. Int.
    ------------------------------------------------------------------------------
    mu         -6.0765e-03  9.197e-03     -0.661      0.509 [-2.410e-02,1.195e-02]
                                   Volatility Model
    =============================================================================
                     coef    std err          t      P>|t|       95.0% Conf. Int.
    -----------------------------------------------------------------------------
    omega      9.9151e-03  6.341e-03      1.564      0.118 [-2.512e-03,2.234e-02]
    alpha[1]       0.1455  5.573e-02      2.611  9.039e-03    [3.626e-02,  0.255]
    beta[1]        0.8168  7.445e-02     10.971  5.253e-28      [  0.671,  0.963]
    =============================================================================

    Covariance estimator: robust
    Time per run: 17.62993399999914 ms

# Conclusion

* We beat Matlab by a factor of about 10 and Python's `arch` package by a factor of about 6, all while delivering estimates that are closer to the benchmark. We also beat R's `rugarch` package by a factor of about 30.
* This is despite the fact that all these packages implement the numerically heavy stuff in C rather than natively.
* This is the _unreasonable effectiveness of Julia_: it walks like Python and runs like C.

#  TODO
* More distributions (NIG, $\alpha$-Stable, ...)
* More GARCH models (APARCH, RiskMetrics, IGARCH, ...)
* Portfolio VaR: Compute the VaR of a portfolio, possibly containing derivatives, via the multivariate Delta-Gamma approximation.
* Made feasible by the results of Broda and Arismendi (Biometrika 2021), under the assumption of a multivariate generalize hyperbolic (MGHyp) distribution.
* This flexible distribution nests, among others, the Normal, Student's $t$, Variance Gamma, Normal Inverse Gaussian, ...
* The scary looking expression on the next slide is already available with my Julia package `QuadraticFormsMGHyp.jl`.

__Theorem__ (Broda and Arismendi, Biometrika 2021) Let $L= a_0+{a}^{T}X+X^{T}{A}X$, where $X\sim \mathrm{MGHyp({{\mu},{C},{\gamma},\lambda,\chi,\psi})}$. Then
\begin{equation}\label{eq:mghcdf}
\mathbb{P}(L\leq l)=\frac{1}{2}-\frac{1}{\pi}\int_{0}^{\infty}\mathrm{Im}\left\{
\Xi_{\lambda}(s,-xs,\chi,\psi)\right\}\frac{\mathrm{d}s}{s},
\end{equation}
where $x= l-a_0-{a}^{T}{\mu}-{\mu}^{T}{A}{\mu}$,
\begin{gather*}
\Xi_{\lambda}(s,t,\chi,\psi)=\frac{k_\lambda\{\chi-2\alpha_2(s)-2\mathrm{i}t,\psi-2\alpha_1(s)\}}{k_\lambda(\chi,\psi)}\rho(s),\\
\alpha_1(s)=\mathrm{i}ks-\frac{1}{2}s^2\sum_{j=1}^d\frac{e_j^2}{1-2\mathrm{i}s\lambda_j},\quad
\alpha_2(s)=-\frac{1}{2}s^2\sum_{j=1}^d\frac{d_j^2}{1-2\mathrm{i}s\lambda_j},\\
\rho(s)=\exp\left(\mathrm{i}sc-s^2 \sum_{j=1}^d\frac{d_je_j}{1-2\mathrm{i}s\lambda_j}\right)\prod_{j=1}^d(1-2\mathrm{i}s\lambda_j)^{-1/2},
\end{gather*}
$\lambda_{j}, j \in\{1,\ldots,d\}$ are the eigenvalues of ${C}^{T}{A}{C}$, ${P}$ is an orthogonal matrix with the corresponding eigenvectors as its columns,
$c= {a}^{T}{\gamma}+ 2 {\mu}^{T}{A}{\gamma}$, $d_j$ and $e_j$ are the individual elements of ${d}={a}^{T}{CP}+2{\mu}^{T}{ACP}$ and ${e}= {\gamma}{ACP}$, and $k={\gamma}^{T}{A}{\gamma}$.

# References
* Bollerslev, T (1986). Generalized autoregressive conditional heteroskedasticity. *Journal of Econometrics* **31**, 307–327.
* Bollerslev, T. & Ghysels, E. (1996). Periodic Autoregressive Conditional Heteroscedasticity. *Journal of Business & Economic Statistics* **14**, 139-151. https://doi.org/10.1080/07350015.1996.10524640.
* Broda, S.A. & Arismendi Zambrano, J. (2021). On quadratic forms in multivariate generalized hyperbolic
random vectors. *Biometrika* **108**, https://doi.org/413-424.0.1093/biomet/asaa067.
* __Broda, S.A. & Paolella, M. (2020). `ARCHModels.jl`: Estimating Arch Models in Julia. Available on SSRN. https://dx.doi.org/10.2139/ssrn.3551503__
* Brooks, C., Burke, S. P., & Persand, G. (2001). Benchmarks and the accuracy of GARCH model estimation. *International Journal of Forecasting* **17**, 45-56. https://doi.org/10.1016/S0169-2070(00)00070-4.
* Engle, R. F. (1982). Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation. *Econometrica* **50**, 987-1007. https://doi.org/10.2307/1912773.
* Engle, R. F., and Manganelli, S. (2004). CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles. *Journal of Business & Economic Statistics* **22**, 367-381. https://doi.org/10.1198/073500104000000370
* Nelson, D.B. (1991). Conditional Heteroskedasticity in Asset Returns: A New Approach. *Econometrica* **59**, 347--370. https://doi.org/10.2307/2938260.

#####
 Thank you! Questions?


# Tutorial

This tutorial is also a Jupyter notebook, which can be found in [the example notebooks directory](https://github.com/annegambrel/xfaster/tree/main/example/notebooks). If you're running the notebook, rather than looking at the documentation produced from it, the links will not work. [Look at the docs for working links](https://annegambrel.github.io/xfaster/notebooks/XFaster_Tutorial.html).

The notebook reads intermediate output npz files from disk. To instead run the pieces starting from maps, first generate the example maps by running the script `xfaster/example/make_example_maps.py`, and set the checkpoint to something other than None.

## The main ingredients of the XFaster code

Before we get into the functions that produce each of the components that gets fed into the equations for bandpower deviations, $q_b$ and the Fisher matrix (Equations 17 and 18 in the [Algorithm section](../algorithm.rst)), let's first talk about how XFaster is structured and what it expects as inputs.

There are two main python modules in XFaster: [xfaster_exec.py](../api.rst#module-xfaster.xfaster_exec) and [xfaster_class.py](../api.rst#module-xfaster.xfaster_class). [xfaster_exec.py](../api.rst#module-xfaster.xfaster_exec) contains two main functions: [xfaster_run](../api.rst#xfaster.xfaster_exec.xfaster_run) and [xfaster_submit](../api.rst#xfaster.xfaster_exec.xfaster_submit). [xfaster_run](../api.rst#xfaster.xfaster_exec.xfaster_run) calls all of the functions to make XFaster happen (all located in [xfaster_class.py](../api.rst#module-xfaster.xfaster_class)) in the order they need to happen. [xfaster_submit](../api.rst#xfaster.xfaster_exec.xfaster_submit) takes arguments for submitting the job to a queue. XFaster is not highly parallelized. However, significant speed-up is gained from using more cores in the simulation pseudo-spectrum calculation through under-the-hood use of OMP. Therefore, if you're starting from a checkpoint after "sims", no significant speed-up will be gained when using more cores.

There are a few other modules you might interact with:

* [parse_tools.py](../api.rst#module-xfaster.parse_tools): contains a bunch of tools for converting between data structures, especially between dictionaries and matrices
* [spec_tools.py](../api.rst#module-xfaster.spec_tools): contains functions for generating and manipulating model power spectra

## Specifying what data to use

The top level module you interact with is [xfaster_exec.py](../api.rst#module-xfaster.xfaster_exec), which takes your arguments, and has some reasonable defaults for any you don't provide. In addition to arguments, you must point the code to the inputs you want to run on. The first of these inputs is maps.

### Maps

The main inputs to the code are maps-- data maps, signal and noise simulations, and masks. Rather than pointing to each map individually, there is a directory structure that the code expects. Its contents look like this:

    <data_root>/
    ├── data_<data_type>
    │   ├── <data_subset1>
    │   │   ├── map_<tag1>.fits
    │   │   ├── ...
    │   │   ├── map_<tagN>.fits
    │   ├── <data_subset2> (same filenames as <data_subset1>)
    │   ├── ....
    │   ├── <data_subsetM>
    ├── signal_<signal_type>
    │   ├── spec_signal_<signal_type>.dat
    │   ├── <data_subset1>
    │   │   ├── map_<tag1>_0000.fits
    │   │   ├── ...
    │   │   ├── map_<tag1>_####.fits
    │   │   ├── ...
    │   │   ├── map_<tagN>_0000.fits
    │   │   ├── ...
    │   │   ├── map_<tagN>_####.fits
    │   ├── ....
    │   ├── <data_subsetM> (same filenames as <data_subset1>)
    ├── noise_<noise_type> (same filenames as signal_<signal_type>)
    ├── masks_<mask_type>
    │   ├── mask_map_<tag1>.fits
    │   ├── ...
    │   ├── mask_map_<tagN>.fits
    [[optional:]]
    ├── foreground_<foreground_type_sim> (same filenames as signal_<signal_type>)
    ├── templates_<template_type>
    │   ├── template1 (same filenames as data_<data_type>)
    │   ├── template2 (same filenames as data_<data_type>)
    └── reference_<reference_type>
        ├── reference1 (same filenames as data_<data_type>)
        └── reference2 (same filenames as data_<data_type>)

Each of `data`, `signal`, `noise`, and `mask` has a top level directory with a preordained, fixed prefix (`data`, `signal`, `noise`, `mask`) and then some suffix specified by the user which is appended with an underscore. So, for example, to run XFaster on the example set of maps, I need to specify in my arguments: `data_type=raw`, `signal_type=synfast`, `noise_type=gaussian`, `mask_type=rectangle`.

All maps must be HEALPIX fits files. Data maps must be named `map_<tag>.fits`; simulated signal and noise maps must be named `map_<tag>_####.fits`, where #### is any length numerical tag indicating a sim realization; and masks are labeled `mask_map_<tag>.fits`.

If using `pol=True`, masks must be 3 x Npix I,Q,U maps. You can use a different mask for intensity and polarization, in which case your mask must be 2 x Npix, where the first row is used for T and the second for P.

To indicate which maps you want, pass the argument `data_subset` a glob-parseable path relative to the top level data directory-- in this case, `data_raw`. [Glob](https://docs.python.org/3/library/glob.html) works just like the unix shell does for matching file paths, so it is easy to test in advance which maps you're going to get. Just do `ls <path_to_data_maps>/full/*150*` and check that all the maps you want to use are listed. Then, you should set `data_subset` to `full/*150*`, for example.

This is all implemented in `_get_data_files()` (called by [get_files()](../api.rst#xfaster.xfaster_class.XFaster.get_files)) like so:
```python

# find all map files
map_root = os.path.join(data_root, "data_{}".format(data_type))
map_files = []
data_subset = data_subset.split(",")
for f in np.atleast_1d(data_subset):
    files = glob.glob(os.path.join(map_root, "{}.fits".format(f)))
    if not len(files):
        raise OSError("Missing files in data subset {}".format(f))
    map_files.extend(files)
data_subset = ",".join(data_subset)
map_files = sorted(map_files)
map_files = [f for f in map_files if os.path.basename(f).startswith("map_")]
map_tags = [
    os.path.splitext(os.path.basename(f))[0].split("_", 1)[1] for f in map_files
]

```
So you've specified which data maps you want to compute power spectra for. To get the corresponding sim maps, the code takes whatever data maps were found matching data_subset, and tries to match those to maps in the sims directory, with the only difference the sim index tag. Here's what that looks like in the function `_get_sim_files()`:
```python
# find all corresponding sims
# for example, when called for signal sims:
#     _get_sim_files(
#         name="signal",
#         root="signal_{}".format(signal_type),
#         subset=signal_subset,
#     )
root = os.path.join(data_root, root)
num_files = None
all_files = []
for f in map_files:
    files = sorted(
        glob.glob(
            f.replace(map_root, root).replace(
                ".fits", "_{}.fits".format(subset)
            )
        )
    )
    nfiles = len(files)
    if not nfiles:
        raise OSError("Missing {} sims for {}".format(name, f))
    ...
    all_files.append(files)
```

It's easy to get errors at the reading files step. XFaster requires that all your maps have the same number of sims, though you are allowed to have a different from of signal and noise sims. If you do not want to use the full set of signal and noise maps in a directory, if you can set the `signal_subset` and `noise_subset` arguments, which default to `'*'`. This glob-parseable tag is applied to the `####` sim index tag. So, if I have a set of sims, eg, `map_95_0000.fits`-`map_95_0099.fits`, as in the example, but for some reason I only want to use the first 50, I can set `signal_subset='00[0-4]*`.

### Other inputs

In addition to maps, the code also requires a config file. This tells the code everything else it needs to know about your data. Here's the config file for the example:

```python

# Band centers in GHz (needed for foreground fitting) for each map tag.
# Tags listed here should be a superset of tags that may be included in a run.
[frequencies]
95 = 94.7
150 = 151.0

# A numpy dictionary containing Bl for each map tag.
# Keys should match those in freqs.
# Missing keys should have a fwhm listed below.
# The value for each key should be a numpy array of dimensions (3, lmax + 1),
# for TT, TP and PP beam windows.
[beam]
beam_product = None
beam_error_product = None

# FHWM in arcmin for each map tag, if using Gaussian beam model.
# Keys should match those in freqs.
# Missing keys should be in the beam product file.
[fwhm]
95 = 41
150 = 29

# Fractional error on the FWHM for each map tag, if using a Gaussian beam model.
# Keys should match those in freqs.
# Missing keys should be in the beam product file.
[fwhm_err]
95 = 0.001
150 = 0.001

# Whether to compute transfer function qb parameters for each map tag.
# Keys should match those in freqs.
[transfer]
95 = true
150 = true
```

This is the full list of options that can be specified in the config file.

## Step by step through the functions called in xfaster_exec
### Setup and checkpoints

Now we'll proceed to stepping through each function in [xfaster_exec](../api.rst#module-xfaster.xfaster_exec). You'll never run the code this way--you'll just call [xfaster_run](../api.rst#xfaster.xfaster_exec.xfaster_run) or [xfaster_submit](../api.rst#xfaster.xfaster_exec.xfaster_submit) with the arguments that then get passed to these functions. But we'll do it this way so we can illustrate some of the intermediate data products as we go.

This tutorial will follow the example script in [xfaster/example](https://github.com/annegambrel/xfaster/tree/main/example). Note however that becasue maps are not stored in the repository, this notebook is instead reading from the intermediate files written to disk.

First, we'll import XFaster and initialize our XFaster class with some arguments.

```python
import xfaster as xf
import matplotlib.pyplot as plt
from matplotlib import colors
import numpy as np
```

```python
X = xf.XFaster(config="../../example/config_example.ini", output_root="../../example/outputs_example", verbose="info",
              output_tag="95x150", checkpoint=None)
```

At the start of an XFaster run, you can specify at what checkpoint to start. The checkpoints are:
```python
checkpoints = [
	"files", # find all the map files
	"masks", # load in masks and compute their pseudo-spectra
	"kernels", # compute mask mode-coupling kernels
	"sims_transfer", # compute pseudo-spectra for signal sims used for computing the transfer function
	"shape_transfer", # load in the model spectrum for the transfer function
	"transfer", # compute the transfer function
	"sims", # compute pseudo-spectra for noise sims and signal sims if they're different from those used for the transfer function
    "beams", # load in the beam window functions, and apply a pixel window function if pixwin=True
    "data", # compute the pseudo-spectra of the data maps
    "template_noise", # compute the pseudo-spectra of template noise simulations, if using Planck dust templates
    "shape", # load in the model spectrum for bandpower computation
	"bandpowers", # do the Fisher iteration to compute bandpowers
	"beam_errors", # start the likelihood over including beam error parameterization
	"likelihood", # compute the parameter likelihoods
]
```

Note that there are options to use different signal simulations for computing the transfer function and for computing other signal terms. This is mainly useful for null tests, as that's the only mode in which signal sims are used for anything other than transfer functions (to compute the expected signal residual that is subtracted from the data pseudo-spectra). If the signal simulations lack sufficient power (eg., for BB), the transfer function may not converge due to lack of measurable power. This is not a problem unique to XFaster, and this option allows the user more flexibility in the choice of sims used for different parts of the algorithm.

Most of the time, `checkpoint` can be set to None, which starts the code from the last checkpoint completed (ie, it successfully wrote its intermediate data to disk). However, if you've made changes to the code or inputs, you may wish to force the code to start from an earlier checkpoint. It will then run that checkpoint and every one after it in the checkpoint tree (given near the top of [the XFaster class](../api.rst#xfaster.xfaster_class.XFaster)).

### get_files

The first function called after initializing the XFaster class is [get_files()](../api.rst#xfaster.xfaster_class.XFaster.get_files). This takes all your arguments specifying what maps to use and finds them all on disk/checks that they're there.

```python
file_opts = {
    "data_root": "../../example/maps_example",
    "data_subset": "full/*95,full/*150",
    "data_type": "raw",
    "noise_type": "gaussian",
    "mask_type": "rectangle",
    "signal_type": "synfast",
}

```

```python
file_vars = X.get_files(**file_opts)
```

```python
file_vars.keys()
```

`file_vars` is a dictionary returned by `get_files` that is also written to disk to finish the checkpoint. There are a lot of things stored in this dictionary that are mainly used internally by the code and aren't useful to an end user. For example, `num_corr` is the number of crosss-correlations of maps, which is used as a dimension to set up data structures later on. But, some of these things might be useful for debuggin. For example, all of the files found using your glob-parseable file tags are stored here, so you can see exactly what maps are being used for data:

```python
file_vars['map_files']
```

Other options you might use here are

* `template_type`, which points to files stored in `templates_<template_type>` for foreground template subtraction
* `template_noise_type`, which points to files stored in `templates_noise_<template_noise_type>` for subtracting a correlated noise component from the template terms
* `reference_type`, which points to files stored in `reference_<reference_type>` for reference signal subtraction for debiasing null tests
* `data_root2`, which is used for null tests. This points to a second full map directory (data, signal and noise sims) for a set of data to be subtracted from the maps in data_root1.
* `signal_type_sim`/`noise_type_sim`/`foreground_type_sim`/`template_type_sim` -- these are tags corresponding to directories `signal_<signal_type_sim>`, `noise_<noise_type_sim>`, `foreground_<foreground_type_sim>`, `template_<template_type_sim>` that are used when the `sim_data` argument is used in the `data` checkpoint. This mode chooses one of the sims for each each component in place of data maps. It adds the signal, noise, and/or foreground or template sims in alm-space. These options are not required to run in `sim_data` mode-- if they are not set, they default to `signal_type_sim=signal_type`, `noise_type_sim=noise_type`, `foreground_type_sim=None` and `template_type_sim=template_type`.

### get_bin_def

Now we run [get_bin_def()](../api.rst#xfaster.xfaster_class.XFaster.get_bin_def) set up a dictionary that tells where the edges of the CMB bins are for each spectrum. We can also choose whether to fit for noise residuals or foregrounds, which we can set to have different bin widths than the CMB bins.

```python
bd = X.get_bin_def(bin_width=25, lmin=2, lmax=500, tbeb=True, foreground_fit=False,
                   residual_fit=True, bin_width_res=100)
```

```python
bd.keys()
```

```python
bd['cmb_tt']
```

This dictionary has a key for each type of spectrum that will be fit, each of which stores the bin edges used for that spectrum fit. Each of these bin edge pairs represents a separate bandpower deviation ($q_b$) that will be fit.

Residuals are fit per map, and by default are fit only for EE and BB, which are constrained to have the same fit parameter. To change this, use the option `res_specs`, which takes a list of the spectra you want to fit residuals for, ie. `["TT", "EE", "BB"]` if you'd like to fit all of the spectra separately.

The last option you might wish to use is `weighted_bins`, which changes the default $\chi_b(\ell)$ binning operator from a tophat to one that weights by $\ell(\ell+1)$.

### get_mask_weights

We have now finished all the set-up. Now it's time to start calculating things. First, we compute the cross spectra of our masks--which will be needed for computing the $K_{\ell \ell^{\prime}}$ mode-coupling matrix--and the $g_\ell$ mode-counting factor. This is done with [get_mask_weights()](../api.rst#xfaster.xfaster_class.XFaster.get_mask_weights).

For the example, we will not apply an empirical correction to $g_\ell$, the calibration of which is discussed in Section 2.3.2 of the [XFaster paper](https://arxiv.org/abs/2104.01172). Otherwise, we would set `apply_gcorr=True`, and the code would look in the masks directory for this correction file for each map. The `reload_gcorr` option is only useful when doing the empirical calibration; it reloads the file, while by-passing the checkpoint tree that is usually performed after `get_mask_weights`.

```python
w_stuff = X.get_mask_weights(apply_gcorr=False, reload_gcorr=False)
```

```python
w_stuff.keys()
```

`w1`, `w2`, and `w4` are the first, second, and fourth moments of the mask. `fsky` is the fraction of sky the mask covers. These are needed for computing the $g_\ell$ mode-counting term, labeled `gmat` in the outputs. `gmat` includes the $2\ell+1$ factor and a correction factor, if desired.

`wls`, contains the cross spectra of all the masks. For this and all other pseudo-spectra, the spectrum is computed just as `map2alm`, and then `alm2cl` healpy routines on the maps. If you're doing a polarized spectrum, you'll get three elements, (intensity, pol, intensity x pol). Since our intensity and pol masks are the same, all three elements are the same, and look like this (plotted as $D_\ell$):

```python
ell = np.arange(501)
lfac = ell * (ell+1) / (2*np.pi)
plt.plot(lfac * w_stuff['wls']['150:150'][0])
plt.ylabel(r'$\frac{\ell(\ell+1)}{2\pi}w_\ell$')
plt.xlabel(r'$\ell$')
```

**Note:** most of the quantities in XFaster are ordered dictionaries with the following nested structure:
1. Spectra (either plain ['tt', 'ee'], etc, or ['cmb_tt', 'cmb_ee', ..., 'fg_tt', 'fg_ee', ..., 'res_tt', ...]
2. Map/map cross. Crosses are indicated with a colon and are in alphabetical order (so 95 comes after 150, for example): ['150:150', '150:95', '95:95']
3. Stuff. Typically an array, though depending on the data structure, it could be a more deeply nested dictionary.

### get_kernels

The next step is to compute the kernels for each mask auto and cross spectrum using the function [get_kernels()](../api.rst#xfaster.xfaster_class.XFaster.get_kernels).  In the MASTER formalism, the kernel couples modes to each other on the sky due to the finite dimensions of the mask that is applied to each map that goes into computing a cross spectrum.  That is to say, the measured power at a particular $\ell$ is a weighted average over several other neighboring modes $\ell^\prime$, and $K_{\ell\ell^\prime}$ is the matrix that applies this weighting.

The kernels are computed from the power spectrum of the mask, computed in the previous step.

Let's plot some of these up.  Because we are using the same mask for all maps, the kernels for each map pair will look identical.

```python
k = X.get_kernels()
print(k.keys())
```

```python
fig, ax = plt.subplots(2, 2, figsize=(12,5.5))
m = ax[0,0].imshow(k['kern']['150:150'], norm=colors.LogNorm(vmax=0.01, vmin=1e-10))
ax[0,0].set_title('kern')
ax[0,0].set_ylabel('$\ell$')
ax[0,1].imshow(k['pkern']['150:150'], norm=colors.LogNorm(vmax=0.01, vmin=1e-10))
ax[0,1].set_title('pkern')
ax[1,0].imshow(k['mkern']['150:150'], norm=colors.LogNorm(vmax=0.01, vmin=1e-10))
ax[1,0].set_title('mkern')
ax[1,0].set_ylabel('$\ell$')
ax[1,0].set_xlabel('$\ell^\prime$')
ax[1,1].imshow(np.abs(k['xkern']['150:150']), norm=colors.LogNorm(vmax=0.01, vmin=1e-10))
ax[1,1].set_title('xkern')
ax[1,1].set_xlabel('$\ell^\prime$')
fig.colorbar(m, ax=ax.ravel().tolist());
```

Note that the shorter dimension ($\ell$) of each kernel goes up to $\ell_{max}$, and the longer dimension ($\ell^\prime$) extends to $2 \ell_{max} + 1$.  The kernels are apodized, so that for any row $\ell$, the kernels are zero for values $\ell^\prime > \ell + \ell_{max}$.  The longer dimension is summed over in computing the $\tilde{\mathcal{C}}_{b\ell}$ terms.

### get_masked_sims

Now we want to get the ensemble average of all of our signal and noise simulations, which we're going to use to calculate the filter transfer function and the noise shape, respectively. This is done with [get_masked_sims()](../api.rst#xfaster.xfaster_class.XFaster.get_masked_sims). The method will also compute the signal cross noise terms, which are used for null tests, where they can contribute significantly to the expected residuals that are subtracted from the data.

This function is doing a very simple thing. For each pair of maps of a given sim index, it

1. Applies the mask.
2. Transforms the maps into $a_{\ell m}$s using the healpy routine `map2alm`.
3. Transforms those into $\tilde{\mathcal{C}}_\ell$s for that pair using the healpy routine `alm2cl`.
4. Adds the $\tilde{\mathcal{C}}_\ell$s to a running average for that particular map cross and spectrum.

It only does `map2alm` once per map and caches the result for use in other cross spectra since this is the slowest step in the function.

Below, we call the function with `transfer=True`-- this tells the code to use the sims specified in `signal_type_transfer`, which defaults to being `signal_type` if the former is not specified.

```python
sims = X.get_masked_sims(transfer=True)
print(sims.keys())
```

The resulting outputs are:

* `cls_signal`: the average of the signal-only cross spectra, used to compute the transfer function
* `cls_noise`: the average of the noise-only cross spectra, used as the noise model, $N_\ell$
* `cls_sim`: the average of the signal+noise spectra, where signal and noise maps are added in $a_{\ell m}$s and thus the spectra include SxN terms.
* `cls_med`: the median of the signal+noise spectra-- this is mainly used for debugging potential biases seen in the pipeline

The rest of the spectra are not symmetrized. For all previously listed spectra, the result is the average of map i x map j and map j x map i, which matters for off-diagonal spectra: TE, TB, EB. Below, we preserve the individual cross spectra as they are needed for the null test model. In this model, at each Fisher iteration, the noise residual fit (per map and per residual spectrum type) is then used to adjust the expectation spectrum subtracted from the data. The expectation spectrum subtracted from, eg, map 1 T x map 2 E is $S_1^T\times S_2^E + S_1^T \times N_2^E + N_1^T \times S_2^E + N_1^T \times N_2^E$. Each of the last three terms is subject to misestimation of $N$, so we account for that by scaling the following terms by the appropriate noise residual fit.

* `cls_res["nxn0"]`: the average spectrum for noise i x noise j
* `cls_res["nxn1"]`: the average spectrum for noise j x noise i
* `cls_res["sxn0"]`: the average spectrum of signal i x noise j
* `cls_res["sxn1"]`: the average spectrum of noise j x signal i
* `cls_res["nxs0"]`: the average spectrum of noise i x signal j
* `cls_res["nxs1"]`: the average spectrum of signal j x noise i

These spectra have all the effects of the masking, filtering, and beam (for signal sims) included. Let's compare a couple of them to get a sense for what they look like.

```python
fig, ax = plt.subplots(1, 2)
ax[0].plot(lfac * sims['cls_signal']['tt']['95:95'], label='95 auto')
ax[0].plot(lfac * sims['cls_signal']['tt']['150:150'], label='150 auto')
ax[0].plot(lfac * sims['cls_signal']['tt']['150:95'], label='95 x 150')
ax[0].legend()
ax[0].set_ylabel(r'$D_\ell [\mu \mathrm{K}^2]$')
ax[0].set_xlabel(r'$\ell$')
ax[0].set_title('TT')
ax[1].plot(lfac * sims['cls_signal']['ee']['95:95'], label='95 auto')
ax[1].plot(lfac * sims['cls_signal']['ee']['150:150'], label='150 auto')
ax[1].plot(lfac * sims['cls_signal']['ee']['150:95'], label='95 x 150')
ax[1].legend()
ax[1].set_xlabel(r'$\ell$')
ax[1].set_title('EE')
fig.suptitle('cls_signal')
```

In this example, we use the same idealized transfer function to make both 95 and 150 GHz maps, so their low-$\ell$ signal is similar. However, for 150 GHz we use a 29 arcmin beam, and for 95 GHz, we use a broader 41 arcmin beam. The difference is evident at high $\ell$, where the 150 GHz signal spectra recover more power.

Similarly, we can plot up the noise averages:

```python
fig, ax = plt.subplots(1, 2)
ax[0].plot(lfac * sims['cls_noise']['tt']['95:95'], label='95 auto')
ax[0].plot(lfac * sims['cls_noise']['tt']['150:150'], label='150 auto')
ax[0].plot(lfac * sims['cls_noise']['tt']['150:95'], label='95 x 150')
ax[0].legend()
ax[0].set_ylabel(r'$D_\ell [\mu \mathrm{K}^2]$')
ax[0].set_xlabel(r'$\ell$')
ax[0].set_title('TT')
ax[1].plot(lfac * sims['cls_noise']['ee']['95:95'], label='95 auto')
ax[1].plot(lfac * sims['cls_noise']['ee']['150:150'], label='150 auto')
ax[1].plot(lfac * sims['cls_noise']['ee']['150:95'], label='95 x 150')
ax[1].legend()
ax[1].set_xlabel(r'$\ell$')
ax[1].set_title('EE')
fig.suptitle('cls_noise')
```

In making the example maps, we have set the noise amplitude to be a bit higher for 95 GHz, which is evident in the noise auto-spectra. As expected, the cross-spectra is uncorrelated and mean-zero.

### get_beams

The next component we need for our equations is the beam window function, $B_\ell$, done with [get_beams()](../api.rst#xfaster.xfaster_class.XFaster.get_beams). XFaster does not solve for this-- you have to tell it what it is. You do this in your config.ini file. You can either specify Gaussian FWHM values (in arcmin) for each map tag, or specify a `.npz` file that contains a dictionary of $B_\ell$ vectors per map tag. You can mix and match these as well-- the code will first look for the tags in the dictionary file, and if it doesn't find the tag there, it will look for a FWHM field for that tag.

The only additional option available is `pixwin`, which is default True. This applies an additional pixel window function to your beam window function, corresponding to the $N_{side}$ of the input maps.

```python
beams = X.get_beams(pixwin=True)
print(beams.keys())
```

We could have a different beam for intensity and polarization. For this example, we don't, so each of the spectrum fields for beam is the same. Let's plot them for each map.

```python
for freq in X.map_tags:
    plt.plot(beams['tt'][freq], label=freq)
plt.legend()
plt.ylabel(r'$B_\ell$')
plt.xlabel(r'$\ell$')
```

### get_signal_shape

Let's check in on our progress of components we've computed. We're trying to build everything to make up our $\tilde{\mathcal{C}}^{XY}_{b\ell}$s, which as a reminder are the following quantity (for TT):

\begin{equation}
\tilde{\mathcal{C}}^{XY}_{b\ell} = \sum_{\ell^{\prime}} K_{\ell \ell^{\prime}}^{X Y} F_{\ell^{\prime}}^{X Y} B_{\ell^{\prime}}^{2} \mathcal{C}_{\ell^{\prime}}^{X Y (S)} \chi_{b}\left(\ell^{\prime}\right)
\end{equation}

We have $K_{\ell, \ell'}$ and $B_{\ell}$. For the transfer function calculation, we're going to set $F_\ell$ to 1 so that we measure $q_b$s as the deviation from a uniform transfer function for our simulations. Binning, $\chi_b$ has been chosen. All that's left is the full sky signal shape, $\mathcal{C}_{\ell'}^{XY (S)}$, loaded with [get_signal_shape()](../api.rst#xfaster.xfaster_class.XFaster.get_signal_shape).

For calculating the transfer function, this is just the shape spectrum that went into making our simulations. This spectrum can be specified by setting the argument `signal_transfer_spec` to a file containing the spectrum. If not provided, the code will look in the maps directory for the signal sims for a file labeled `spec_signal_<signal_type>.dat`. The file is expected to look like a CAMB output file, as demonstrated in `make_example_maps.py`, which writes such a file to the proper location in the signal sims directory.

```python
signal_shape = X.get_signal_shape()
print(signal_shape.keys())
```

These will look familiar to you. Note that they are as long as the long dimension of the $K_{\ell\ell'}$, which is 2$\ell_{max}$ + 1. Also note that EB and TB, expected to be zero, are set to a small flat value.

```python
ell2 = np.arange(1001)
lfac2 = ell2 * (ell2 + 1) / (2*np.pi)
fig, ax = plt.subplots(2, 3, figsize=(10,7))
ax = ax.flatten()
for i, (s, spec) in enumerate(signal_shape.items()):
    ax[i].plot(lfac2 * spec)
    ax[i].set_title(s)
    if i in [0,3]:
        ax[i].set_ylabel(r'$D_\ell [\mu\mathrm{K}^2]$')
    if i in [3,4,5]:
        ax[i].set_xlabel(r'$\ell$')
```

### get_transfer

We now have everything we need to compute the transfer function, which is computed per map per spectrum per CMB bin in [get_transfer()](../api.rst#xfaster.xfaster_class.XFaster.get_transfer). As a refresher, we're trying to get $q_b^{transfer}$, which is the same as our original expression for $q_b$ in [Equation 17 of the Algorithm page](https://annegambrel.github.io/xfaster/algorithm.html#equation-qb), except now we set noise=0, transfer function=1, and instead of using data for our observed signal, we use the ensemble average of our signal sims:

\begin{equation}
q_{b}^{transfer}=\frac{1}{2} \sum_{b^{\prime}} \mathcal{F}_{b b^{\prime}}^{-1} \sum_{\ell} (2 \ell+1) g_\ell^k\left[ \left(\tilde{\mathbf{C}}_{\ell}^{-1} \frac{\partial \tilde{\mathbf{S}}_{\ell}}{\partial q_{b^{\prime}}} \tilde{\mathbf{C}}_{\ell}^{-1}\right)\tilde{\mathbf{C}}_{\ell}^{signal}\right]_{kk}
\label{eq:qb_transfer}
\end{equation}

The expression for the Fisher matrix does not change, other than the fact that its constituents are the same as detailed above for the transfer function.

Within the code, `get_transfer` basically has two steps within the function itself, which it performs per map.

1. Load up the $\tilde{\mathcal{C}}_{b\ell}$:
```python
cbl = self.bin_cl_template(map_tag=m0, transfer_run=True)
```
This uses the signal_shape internally that we calculated earlier, `m0` is the map, which is used to select the beam and kernel, and `transfer_run=True` sets the $F_\ell$ term to 1.

2. Run [fisher_iterate()](../api.rst#xfaster.xfaster_class.XFaster.fisher_iterate).
```python
ret = self.fisher_iterate(cbl, m0, transfer_run=True,
                          iter_max=iter_max, converge_criteria=converge_criteria,
                          save_iters=save_iters, ...)
```

We'll talk more in the `get_bandpowers` section about the details that happen from here, but broadly, XFaster takes all the inputs we've calculated and a starting $q_b$ guess (1 for all bins), computes the Fisher matrix, plugs that into the $q_b$ equation to get a new $q_b$, and repeats. Once the maximum of |(qb_new-qb)/qb| < converge_criteria, it stops iterating and saves the result.

One additional check that `get_transfer` does is to look for transfer function values that are negative. If it finds any, it changes that bin value to the average of the orignal value and the next bin's value. This typically happens due to poor choices of binning or too small a number of signal simulations.

Also, Only TT, EE, BB, and TE transfer functions are calculated. EB and TB are computed as
```python
qb['cmb_eb'] = np.sqrt(np.abs(qb['cmb_ee'] * qb['cmb_bb']))
qb['cmb_tb'] = np.sqrt(np.abs(qb['cmb_tt'] * qb['cmb_bb']))
```

```python
transfer = X.get_transfer()
print(transfer.keys())
```

Let's look at a couple transfer functions.

```python
fig, ax = plt.subplots(1, 2, figsize=(10,7), sharey=True)
ax = ax.flatten()
plot_inds = {'95': 0, '150': 1}
for s, spec in transfer.items():
    for m, fl in spec.items():
        ax[plot_inds[m]].plot(fl, label=s.split('_')[-1])
        if 'tt' in s:
            ax[plot_inds[m]].set_title(m)
for i in range(2):
    ax[i].set_xlabel('bin')
for i in [0]:
    ax[i].set_ylabel(r'$F_\ell$')
ax[0].legend()
ax[0].set_ylim(0,1.2)
```

Note that the BB transfer function (and the EB/TB terms computed from it) start to diverge at high multipole. This is due to the lack of signal power in our input spectrum combined with the reduction in power of the beam. For this reason, you might choose to use a model spectrum with more BB power in computing your BB transfer functions.

### get_masked_data

Now comes the step where we compute the data term using [get_masked_data()](../api.rst#xfaster.xfaster_class.XFaster.get_masked_data). This is once again done just by taking all the pseudo-cross-spectra of all the data maps, using `healpy.map2alm` and `healpy.alm2cls`. This is also the step where the cross spectra of foreground templates is performed, if `template_type` is specified.

```python
cls_data = X.get_masked_data()
print(cls_data.keys())
```

```python
fig, ax = plt.subplots(1, 2)
ax[0].plot(lfac * cls_data['cls_data']['tt']['95:95'], label='95 auto')
ax[0].plot(lfac * cls_data['cls_data']['tt']['150:150'], label='150 auto')
ax[0].plot(lfac * cls_data['cls_data']['tt']['150:95'], label='95 x 150')
ax[0].legend()
ax[0].set_ylabel(r'$D_\ell [\mu \mathrm{K}^2]$')
ax[0].set_xlabel(r'$\ell$')
ax[0].set_title('TT')
ax[1].plot(lfac * cls_data['cls_data']['ee']['95:95'], label='95 auto')
ax[1].plot(lfac * cls_data['cls_data']['ee']['150:150'], label='150 auto')
ax[1].plot(lfac * cls_data['cls_data']['ee']['150:95'], label='95 x 150')
ax[1].legend()
ax[1].set_xlabel(r'$\ell$')
ax[1].set_title('EE')
fig.suptitle('cls_data')
```

There are several options for manipulating the constructed data before they are passed to the estimator.

* `template_alpha`: not used for null tests.  This is a dictionary of scalar values, keyed by map tag, to scale foreground templates to be subtracted from the data.  The subtraction is done in alm-space for each tag that is included in the dictionary.
* If `reference_type` is set at the `files` checkpoint, then some reference signal is subtracted from each half of a null split.  Typically the reference would be something like _Planck_ half-mission datasets, where the _Planck_ instrument noise between the two sets is uncorrelated, such that the cross spectrum of the two reference signals is independent of _Planck_ instrument noise.
* If `template_noise_type` is set at the `files` checkpoint, then an ensemble of noise simulations is used to construct an estimate of the instrument noise contribution to the template subtraction and remove it from the data.  Typically this is an ensemble of _Planck_ FFP10 simulations, which by construction introduces a slight correlation in the noise between template halves.  This option accounts for that correlation.
* `template_specs`: A list of spectra for which the template subtraction is applied.  By default, this is done for all spectra that the XFaster estimator is computing.  However, when excluding TT/TE signal from the likelihood, it is necessary to avoid subtracting foreground templates from the TT/TE spectra as well.

Now, computing things for data is typically the final thing you'll be wanting to do. The first way you'll probably run the code is on simulations to make sure things make sense.  The options for doing this are explained below.

### Sim options for get_masked_data

There are several options to `get_masked_data()` that dictate how to build a simulated data set from an ensemble of components on disk.  Note that these options may be passed to this function differently from `xfaster_run`.  We'll go through all of them here.

First, there are three options that determine whether any simulated dataset is to be used at all in place of data.  If these are all `False`, then the data maps are loaded from disk as usual.

* `ensemble_mean`: use `cls_sim` in place of the data spectra. This quantity, computed in [the previous sims step](#get_masked_sims) is the average of the signal+noise sims.
* `ensemble_median`: use `cls_med`, the median of the signal+noise sims, in place of the data.
* `sim`: if this option is `True`, the remaining options below are used to determine how the simulated dataset is constructed.  In `xfaster_run`, this option is called `sim_data`.

The following options control the construction of the dataset when `sim=True`.

* `components`: A list of components to include in the simulation.  These may include `signal`, `noise`, `foreground` or `template`, and require the appropriate `<component>_type_sim` variable to have been set in `get_files()` above.  In `xfaster_run`, this option is called `sim_data_components`.
* `index`: A dictionary of indices keyed by component name that determine which file from each ensemble to use to construct the data.  For example, `{"signal": 0, "noise": 10}` uses difference sim indices for the signal and noise components.  A `default` key sets the sim index to use for any component not included in the dictionary.  If no default is given, then `0` is assumed.  NB: in the `xfaster_run` function, this dictionary is constructed from the arguments `sim_index_default`, `sim_index_signal`, `sim_index_noise` and `sim_index_foreground`.
* `r`: If this is not `None`, then the `signal` component of the simulated data is constructed from a linear combination of a scalar CMB term and a tensor CMB term, with the latter scaled by the value of `r`.  This option was used for SPIDER to run the ensemble of simulations needed for the Feldman-Cousins analysis, and was written to allow for constructing many different $r$-input maps in memory. For this option to work, the `signal_type_sim` argument must be set to `"r"`, in which case the `get_files()` function searches for an ensemble of scalar sims in the subdirectory `signal_r0` and tensor sims in the subdirectory `signal_r1tens`.  In `xfaster_run`, this option is called `sim_data_r`.
* `qb_file`: If this option is not `None`, then it points to a `bandpowers.npz` file on disk, which contains noise residual bandpowers that can be used to scale the noise component of the simulated data.  In `xfaster_run`, this option is called `qb_file_data`.  NB: a similar option, `qb_file_sim`, can be passed to `get_masked_sims()` to instead apply the residual correction to the noise ensemble that goes into the covariance model (e.g. to test for bias or in single-map mode where fitting for noise residuals doesn't work).
* `template_alpha_sim`: A dictionary similar to `template_alpha` above that handles the _addition_ of foreground templates to the simulated dataset.  In `xfaster_run`, this dictionary is constructed using the arguments `template_alpha_sim` (a list of floats) and `template_alpha_tags_sim` (a list of map tags for each float).
* `save_sim`: If True, the constructed simulation is stored to disk to avoid rebuilding it later.  The filename is constructed as `data_<...>_xcorr.npz` where a set of tags is built from the components that are included in the simulation.  The output `bandpowers.npz` file corresponding to each such simulation includes the same set of tags in the filename.  This file naming scheme allows running multiple simulations in parallel without rerunning intermediate steps or creating IO conflicts on disk.  In `xfaster_run`, this option is called `save_sim_data`.

For the example case, we won't use any of these options. They can be super useful for forming expectations for your data outputs, though!

### get_bandpowers

Now we put it all together!  The function [get_bandpowers()](../api.rst#xfaster.xfaster_class.XFaster.get_bandpowers) works just like `get_transfer`, with the transfer function terms no longer set to unity, and instead constructed from the $q_b$s computed by `get_transfer`.

#### Constructing the model spectrum

The first step is to construct the $\tilde{\mathcal{C}}_{b\ell}$'s including all of the additional components in the model. This is done with [bin_cl_template()](../api.rst#xfaster.xfaster_class.XFaster.bin_cl_template).

```python
cbl = X.bin_cl_template(map_tag=None, transfer_run=False, use_precalc=False)
```

Let's plot up each set of components individually.  First, the CMB component, which should look familiar.  Each color represents the $\tilde{\mathcal{C}}_{b\ell}$ for a single bin.  The black lines show the total shape of each component in $\ell$ if the $q_b$ applied to each bin is set to 1 (ie, our signal model is perfectly correct).  This is what we call the *model spectrum* that we are fitting to.  Notice that the EE and BB mixing terms have very broad bins and are very low amplitude.  This is due to the shape of the $_{-}K_{\ell\ell^\prime}$ kernel as a function of $\ell$.  These shapes will look different for each map cross, due to differences in each transfer function and beam.

```python
fig, axs = plt.subplots(2, 4, figsize=(14,7), sharex=True)
axs = axs.ravel()
ell = np.arange(501)
ellfac = ell * (ell + 1) / 2.0 / np.pi
comps = ['cmb_tt', 'cmb_te', 'cmb_ee', 'cmb_ee_mix', 'cmb_eb', 'cmb_tb', 'cmb_bb', 'cmb_bb_mix']
for ax, comp in zip(axs, comps):
    ax.set_title('150 x 150 {}'.format(comp))
    cbl1 = cbl[comp]['150:150']
    d = cbl1.sum(axis=0)
    ax.plot(d * ellfac, 'k')
    for d in cbl1:
        ax.plot(d * ellfac)
for i in [0,4]:
    axs[i].set_ylabel('$D_\ell$')
for i in range(4,8):
    axs[i].set_xlabel('$\ell$')
plt.tight_layout()
```

The next model component that we include accounts for residual noise in the auto-spectra.  These $\tilde{\mathcal{C}}_{b\ell}$ terms do not include any transfer functions, beams or mode mixing kernels.  They are derived from the *simulated* noise spectra $\tilde{N}_\ell$.  These terms act as corrections to the noise model by adjusting the auto-spectrum noise components to agree with the cross terms.

For the example, we fit only for the EE and BB components, and we require them to be the same.  That is, each $q_b^{res,EEBB}$ applies to both $\tilde{\mathcal{C}}_{b\ell}^{EE}$ and $\tilde{\mathcal{C}}_{b\ell}^{BB}$ here.

The residual signal model is thus:

\begin{equation}
\tilde{\mathbf{S}}_\ell^{res,ij} = \delta_{ij}\,
\begin{bmatrix}
0 & 0 & 0 \\
0 & \sum_b q_b^{EEBB} \tilde{\mathcal{C}}_{b\ell}^{EE} & 0 \\
0 & 0 & \sum_b q_b^{EEBB} \tilde{\mathcal{C}}_{b\ell}^{BB} \\
\end{bmatrix}_{\,res,ij}
\label{eq:signal_res}
\end{equation}

and the derivatives are

\begin{equation}
\frac{\partial \tilde{\mathbf{S}}_\ell}{\partial q_b^{res,ij,EEBB}} = \delta_{ij}\,
\begin{bmatrix}
0 & 0 & 0 \\
0 & \tilde{\mathcal{C}}_{b\ell}^{EE} & 0 \\
0 & 0 & \tilde{\mathcal{C}}_{b\ell}^{BB} \\
\end{bmatrix}_{\,res,ij}
\label{eq:signal_res_deriv}
\end{equation}

```python
fig, axs = plt.subplots(1, 2, sharex=True, figsize=(7,3.5))
axs = axs.ravel()
ell = np.arange(501)
ellfac = ell * (ell + 1) / 2.0 / np.pi
comps = ['res_ee', 'res_bb']
for ax, comp in zip(axs, comps):
    ax.set_title('150 x 150 {}'.format(comp))
    cbl1 = cbl[comp]['150:150']
    d = cbl1.sum(axis=0)
    ax.plot(d * ellfac, 'k')
    for d in cbl1:
        ax.plot(d * ellfac)
for i in [0]:
    axs[i].set_ylabel('$D_\ell$')
for i in range(2):
    axs[i].set_xlabel('$\ell$')
plt.tight_layout()
```

Next we add our actual noise simulations to the model.  The residual noise terms discussed above are designed to account for inaccuracies in this model. For both the fiducial noise model and the noise residuals, we assume 0 average noise in cross-spectra, and so only add them to the auto-terms of the model.

We often find that the Fisher iterations have trouble converging due to numerical errors that cause the covariance to become singular. To prevent this from happening, we also include a term we call "conditioning noise" along the TT/EE/BB diagonals as well.  The conditioning noise is modeled as constant in $\ell$, where the EE and BB diagonals are set to be `cond_noise` and TT is set to be 10*`cond_noise`. The typical conditioning noise is very small (1e-5 or so), and it is automatically adjusted to be the lowest level possible while still having the algorithm converge. In the final Fisher iteration, the conditioning noise is dropped.

To plot up both of these model components, along with all of the terms discussed above, let's call an internal function that computes the model spectrum for us, given the $\tilde{\mathcal{C}}_{b\ell}$ terms and some model parameters $q_b$: [get_model_spectra()](../api.rst#xfaster.xfaster_class.XFaster.get_model_spectra).

```python
from collections import OrderedDict
# construct a dummy qb array to compute input model
qb = OrderedDict([(k, np.ones(len(v))) for k, v in X.bin_def.items()])
cls_model = X.get_model_spectra(qb, cbl, delta=True, cls_noise=X.cls_noise, cond_noise=1e-5)

fig, axs = plt.subplots(2, 3, figsize=(11,7))
axs = axs.ravel()
ell = np.arange(501)
ellfac = ell * (ell + 1) / 2.0 / np.pi
xname = '150:150'
for comp in ['cmb', 'noise', 'res', 'cond', 'total']:
    for ax, spec in zip(axs, ['tt', 'ee', 'bb', 'te', 'eb', 'tb']):
        stag = '{}_{}'.format(comp, spec)
        if stag not in cls_model:
            ax.plot(0, 0, label=comp)
            continue
        if comp in ['res', 'total']:
            ax.plot(ellfac * cls_model[stag][xname], label=comp, linestyle='--')
        else:
            ax.plot(ellfac * cls_model[stag][xname], label=comp)
        ax.set_title('150 x 150 {}'.format(spec))
        if spec in ['ee', 'bb']:
            ax.set_ylim(-0.01, 0.175)
axs[0].legend()
for i in [0,3]:
    axs[i].set_ylabel('$D_\ell$')
for i in range(3,6):
    axs[i].set_xlabel('$\ell$')
plt.tight_layout()
```

```python
fig, axs = plt.subplots(2, 3, figsize=(11,7))
axs = axs.ravel()
ell = np.arange(501)
ellfac = ell * (ell + 1) / 2.0 / np.pi
xname = '150:95'
for comp in ['cmb', 'noise', 'res', 'cond', 'total']:
    for ax, spec in zip(axs, ['tt', 'ee', 'bb', 'te', 'eb', 'tb']):
        stag = '{}_{}'.format(comp, spec)
        if stag not in cls_model or xname not in cls_model[stag]:
            ax.plot(0, 0, label=comp)
            continue
        ax.plot(ellfac * cls_model[stag][xname], label=comp)
        ax.set_title('150 x 95 {}'.format(spec))
axs[0].legend()
for i in [0,3]:
    axs[i].set_ylabel('$D_\ell$')
for i in range(3,6):
    axs[i].set_xlabel('$\ell$')
plt.tight_layout()
```

Note that neither conditioning noise nor residual terms are present in the cross-spectra.

#### Computing the Fisher matrix

Ok! Now that we have all the pieces in place, all that's left is to do some matrix math and iterate over the equations for [q_b](https://annegambrel.github.io/xfaster/algorithm.html#equation-qb) and the [Fisher matrix]([qb](https://annegambrel.github.io/xfaster/algorithm.html#equation-qb) until the results converge.  The two equations are constructed in the function [fisher_calc()](../api.rst#xfaster.xfaster_class.XFaster.fisher_calc), which is called iteratively by [fisher_iterate()](../api.rst#xfaster.xfaster_class.XFaster.fisher_iterate) until convergence is reached.

In order to make all the quantities we've computed look like matrices, we have two utility functions, [dict_to_dmat()](../api.rst#xfaster.parse_tools.dict_to_dmat) and [dict_to_dsdqb_mat()](../api.rst#xfaster.parse_tools.dict_to_dsdqb_mat), that convert the dictionaries `Dmat1` (corresponding to $\mathbf{\tilde{C}}_\ell$ in the equations) and `dSdqb_mat1_freq` ($\partial \mathbf{\tilde{S}}_\ell / \partial q_b$) into matrices.

The `Dmat1` matrix has shape `(3 * num_maps, 3 * num_maps, lmax + 1)` and contains the `total` model terms for each map cross spectrum, ordered as [shown here](https://annegambrel.github.io/xfaster/algorithm.html#equation-dell).  The `dSdqb_mat1_freq` matrix has shape `(3 * num_maps, 3 * num_maps, nbins, lmax + 1)` and includes the CMB, residuals and frequency-corrected foreground components for all the crosses. We then compute the quantity

\begin{equation}
\mathbf{\tilde{C}}^{-1} \frac{\partial \mathbf{\tilde{S}}}{\partial q_b} \mathbf{\tilde{C}}^{-1}
\end{equation}

using the following code:

```python
Dinv = np.linalg.inv(Dmat1.swapaxes(0, -1)).swapaxes(0, -1)
eye = np.eye(len(gmat))
mat1 = np.einsum('ij...,jk...->ik...', eye, Dinv)
mat2 = np.einsum('klm...,ln...->knm...', dSdqb_mat1_freq, Dinv)
mat = np.einsum('ik...,knm...->inm...', mat1, mat2)
```

The first line computes the inverse of $D$, ell-by-ell along the first two dimensions.  The next two lines are necessary due to some quirk of memory access in python; this is just multiplying the first two dimensions of `Dmat1` by the identity matrix.  The last two lines do the matrix multiplication along the first two dimensions of each of the matrices.

Now, we apply the $\mathbf{g}_\ell$ mode-counting factor, and at the same time take the trace and sum over ell, computing everything needed for $q_b$ except $\Sigma_b \mathcal{F}_{b b^{\prime}}$:
```python
qb_vec = np.einsum("iil,ijkl,jil->k", gmat, mat, Dmat_obs) / 2.0
```
The Fisher matrix is:
```python
fisher = np.einsum("iil,ijkl,jiml->km", gmat, mat, dSdqb_mat1_freq) / 2
```

Finally, we get our updated estimate of $q_b$ and its covaraince, the inverse Fisher matrix, and then convert the former array into a readable dictionary:
```python
qb_vec = np.linalg.solve(fisher, qb_vec)
inv_fish = np.linalg.solve(fisher, np.eye(len(qb_vec)))
qb_vec = pt.arr_to_dict(qb_vec, qb)
```

With each iteration, we compare the new `qb_vec` quantity to the previous one, until the maximum fractional change in any element of the array is less than the convergence criterion (0.005 is typical).

#### Constructing final bandpowers

To construct bandpowers, we need to convert the $q_b$ parameters to spectrum bandpowers. The function that handles this is [do_qb2cb()](../api.rst#xfaster.xfaster_class.XFaster.do_qb2cb) This is done by first constructing window functions for each $q_b$ parameter (using [fisher_calc()](../api.rst#xfaster.xfaster_class.XFaster.fisher_calc) with the option `windows=True`), then using that to build a matrix that rotates $q_b$ into bandpowers.  For this example, the quantity we want in the end should be in units of $\mathcal{D}_\ell = \ell (\ell + 1) C_\ell / 2\pi$. You can also return bandpowers in units of $C_\ell$ using `return_cls=True`, which will change the rotation matrix.  Explicitly, we have for the $\mathcal{D}_b$ bandpowers:

\begin{equation}
\mathcal{D}^{XY}_b = \sum_B q^{XY}_B \frac{\partial\langle \mathcal{D}^{XY}_b\rangle}{\partial q^{XY}_B} =
\sum_B q^{XY}_B \frac{\sum_\ell \mathcal{N}_\ell W^{XY(q)}_{b\ell} \chi^{XY}_{B \ell} \mathcal{D}^{XY(S)}_\ell} {\sum_\ell \mathcal{N}_\ell W^{XY(q)}_{b\ell} }\,,
\end{equation}

where the normalization function is

\begin{equation}
\mathcal{N}_\ell = \frac{1}{2}\frac{(2\ell + 1)}{\ell (\ell + 1)}\,,
\end{equation}

and the matrix of derivatives is block-diagonal in the spectrum component $XY$. Similarly, the errors on each bandpower are:

\begin{equation}
\Delta\mathcal{D}_b = \left[\sum_{BB^\prime} \mathcal{F}^{-1}_{BB^\prime} \frac{\partial\langle \mathcal{D}_b\rangle}{\partial q_B} \frac{\partial\langle \mathcal{D}_{b}\rangle}{\partial q_{B^\prime}}\right]^{1/2}\,,
\end{equation}

and the window functions for each $\mathcal{D}_b$ are:

\begin{equation}
W_{b\ell}^{XY(\mathcal{D})} = \sum_B W_{B\ell}^{XY(q)} \frac{\partial\langle \mathcal{D}^{XY}_b\rangle}{\partial q^{XY}_B} \,,
\end{equation}

with normalization $\sum_\ell \mathcal{N}_\ell W_{b\ell}^{(\mathcal{D})} = 1$.  The bandpowers, errors, $q_b$ window functions and bandpower window functions are all stored in the bandpowers output file.  More details on the bandpower window functions can be found in Section 2.5 of the [XFaster paper](https://arxiv.org/abs/2104.01172).

#### Looking at results

Now we'll just run the function that wraps up all these internal functions, [get_bandpowers()](../api.rst#xfaster.xfaster_class.XFaster.get_bandpowers).

```python
bp = X.get_bandpowers(converge_criteria=0.005, iter_max=200, save_iters=True, cond_noise=1e-5, cond_criteria=5e3)
```

In the plot below, we'll plot up the maximum fractional change in $q_b$ with iteration. This information is printed in the logs, but it's helpful to visualize on a plot. You can see it bounce around early on before settling down and ultimately stopping once it gets to the desginated criterion.

```python
import glob
iters = sorted(glob.glob('../../example/outputs_example/95x150/bandpowers_iter*'))
### Figure for seeing how convergence looks in fqb
fig_fqb, ax_fqb = plt.subplots()
ax_fqb.axhline(0.005, label='convergence criterion', color='C1')
ax_fqb.set_title('Max fractional change in qb vs iter')
ax_fqb.set_ylabel('Maximum absolute fqb')
ax_fqb.set_xlabel('Iteration')
fqb = np.zeros(len(iters))
for i, bp0 in enumerate(iters):
    b = xf.load_and_parse(bp0)
    fqb[i] = np.max(np.abs(b['fqb']))
ax_fqb.plot(fqb, marker='o')
ax_fqb.legend()
ax_fqb.set_yscale('log')
```

Now let's see how the total model (summing up the $q_b\mathcal{C}_{b\ell}$ terms for every component) to see how it converges to fit the data. Within a few iterations, it's quite close.

```python
### Figure for looking at how sum(qbCbl) changes with iteration to match data
figtmp = plt.figure()
tmpplot = plt.imshow(np.zeros((2,2)), cmap=plt.cm.get_cmap('viridis'),vmin=0, vmax=len(iters))
plt.close(figtmp)
fig_dat, ax_dat = plt.subplots(2,3, figsize=(15,10))
ax_dat = ax_dat.flatten()
specs = ['tt', 'ee', 'bb', 'te', 'tb', 'eb']
colors = plt.cm.get_cmap('viridis', len(iters)).colors
for s, spec in enumerate(specs):
    ax_dat[s].set_title(spec)
for i, bp0 in enumerate(iters):
    b = xf.load_and_parse(bp0)
    for s, spec in enumerate(specs):
        if i == 0:
            # data cls don't change with iter- plot once.
            ax_dat[s].plot(b['cls_obs'][spec]['150:95'] * lfac, color='gray', label='data')
        ax_dat[s].plot(b['cls_model']['total_'+spec]['150:95'] * lfac, color=colors[i])
for i in [0, 3]:
    ax_dat[i].set_ylabel(r'$D_\ell$')
for i in range(3,5):
    ax_dat[i].set_xlabel(r'$\ell$')
ax_dat[0].legend()
fig_dat.colorbar(tmpplot, label='Iteration', ax=ax_dat.ravel().tolist(),
                 orientation='horizontal', shrink=0.5)

fig_dat.suptitle(r'95$\times$150 GHz total power fit')
```

Now let's look at just the $q_b$ values for each of the different component and see how they converge as a function of iteration and bin. The lowest bins tend to have the biggest moves, but for all components, everything gets close to its final value within a handful of iterations.

```python
### Figure for looking at how qb for each bandpower changes with iteration
figtmp = plt.figure()
tmpplot = plt.imshow(np.zeros((2,2)), cmap=plt.cm.get_cmap('viridis'),vmin=0, vmax=len(iters))
plt.close(figtmp)
specs = ['tt', 'ee', 'bb', 'te', 'tb', 'eb']
colors = plt.cm.get_cmap('viridis', len(iters)).colors
scatter = 1. / 50.
for comp in ['cmb']:
    fig_dat, ax_dat = plt.subplots(3,2, figsize=(20,15))
    fig_dat.suptitle(comp+r' $q_b$ vs. iteration')
    ax_dat = ax_dat.flatten()
    for s, spec in enumerate(specs):
        ax_dat[s].set_title(spec)
        ax_dat[s].axhline(1, color='gray', alpha=0.7)
    for i, bp0 in enumerate(iters):
        b = xf.load_and_parse(bp0)
        for s, spec in enumerate(specs):
            ax_dat[s].scatter(np.arange(20) + scatter * i,
                              b['qb'][comp+'_'+spec], color=colors[i], s=4)
    for i in [0, 2, 4]:
        ax_dat[i].set_ylabel(r'$q_b$')
    for i in range(4,5):
        ax_dat[i].set_xlabel('bin')

    plt.subplots_adjust(wspace=0.05, hspace=0.01)
    fig_dat.colorbar(tmpplot, label='Iteration', ax=ax_dat.ravel().tolist(),
                     orientation='horizontal', shrink=0.5)

```

Now we'll look at the noise residuals that were fit. In the script where we make the example maps, we made the "data" maps have 15% less noise in them than the those used for the noise model. So, we should expect the total noise, $(1+n_b)\left<N\right>$, to be $0.85^2\left<N\right>$, where the square comes because we're fitting in the power spectrum. So, $n_b$s should converge to about -0.28.

```python
# Now plot residuals
fig_dat, ax_dat = plt.subplots(1,2, figsize=(10,8))
fig_dat.suptitle(r'res $q_b$ vs. iteration')
ax_dat = ax_dat.flatten()
maps = [x for x in bp['qb'] if 'res' in x]
for s, m0 in enumerate(maps):
    ax_dat[s].set_title(m0)
    ax_dat[s].axhline(0, color='gray', alpha=0.7)
    ax_dat[s].set_ylim(-2,2)
for i, bp0 in enumerate(iters):
    b = xf.load_and_parse(bp0)
    for s, m0 in enumerate(maps):
        ax_dat[s].scatter(np.arange(5) + scatter * i,
                          b['qb'][m0], color=colors[i], s=4)
for i in [0]:
    ax_dat[i].set_ylabel(r'$q_b$')
for i in range(2):
    ax_dat[i].set_xlabel('bin')

plt.subplots_adjust(wspace=0.05, hspace=0.01)
fig_dat.colorbar(tmpplot, label='Iteration', ax=ax_dat.ravel().tolist(),
                 orientation='horizontal', shrink=0.5)
```

Now that everything has converged, let's plot up what the final bandpowers and error bars look like. XFaster saves the binned bandpowers as $D_\ell$s by default, though they are labeled `cb`. The error bars are in `dcb`, and those are also computed without sample variance in `dcb_nosampvar`. Covariance is in `cov`, and it also has a no sample variance version, `cov_nosampvar`.

To calculate error bars and covariances without sample variance, XFaster just does one final calculation of the Fisher matrix (after everything has converged), with all the $q_b$s set to a very small number, thereby nulling out the signal. The error bars and covariance without sample variance come from that Fisher matrix.

```python
for comp in ['cmb']:
    fig, ax = plt.subplots(3,2, figsize=(20,15))
    ax = ax.flatten()
    fig.suptitle('{} final estimated bandpowers'.format(comp))
    for s, spec in enumerate(specs):
        sn = '{}_{}'.format(comp, spec)
        ax[s].set_title(spec)
        ax[s].plot(lfac * signal_shape[sn][:501], 'k-', label='Model')
        ax[s].errorbar(bp['ellb'][sn], bp['cb'][sn], bp['dcb'][sn], label='sampvar')
        ax[s].errorbar(bp['ellb'][sn]+2, bp['cb'][sn], bp['dcb_nosampvar'][sn], label='no sampvar')
    ax[0].legend()
    for i in [0, 2, 4]:
        ax[i].set_ylabel(r'$D_\ell$')
    for i in [4,5]:
        ax[i].set_xlabel(r'$\ell$')
```

The error bars are just the diagonal of the covariance matrix. Let's look at the covariance matrix with and without sample variance.

```python
from matplotlib.colors import LogNorm
norm = LogNorm(vmin=10, vmax=1e4)
fig, ax = plt.subplots(1, 2, figsize=(15,10))
fig.suptitle('Covariance')
p = ax[0].imshow(np.abs(bp['cov']), norm=norm)
ax[0].set_title('With sample variance')
ax[1].imshow(np.abs(bp['cov_nosampvar']), norm=norm)
fig_dat.colorbar(p, label='Covariance', ax=ax.ravel().tolist(),
                 orientation='horizontal', shrink=0.5)
ax[1].set_title('Without sample variance')
```

This is a 120 x 120 bin matrix, with each of the 20 CMB bins for each spectrum in order (TT, EE, BB, TE, EB, TB).

### get_likelihood

Now we can compute parameter likelihoods with [get_likelihood()](../api.rst#xfaster.xfaster_class.XFaster.get_likelihood). All of the things we've constructed so far are used, except instead of parameterizing the model with $q_b$ bandpower deviations, we set those to 1, and then make the model a function of cosmological parameters. For the example, we will just use $r$ as our parameter. We then compute the likelihood for the data for each step in a Monte Carlo sampler, where we just vary $r$. For this example, since we're only varying one paramter, we'll do a brute force search:

```python
loglike = X.get_likelihood(qb=bp['qb'], inv_fish=bp['inv_fish'], mcmc=False)
```

```python
plt.plot(loglike[0], np.exp(loglike[1]-loglike[1].max()))
plt.ylabel('Likelihood')
plt.xlabel(r'$r$')
```

We can see that we recover something pretty close to 1, our input $r$ value for the example. If we ran this for many signal+noise sims, we should get precisely 1 in the average.

If you instead are using mcmc=True, the likelihoods file written to disk will have a field called "samples". Histogramming that up, or using a program like `getdist`, especially helpful for multiple parameters, will give you the posteriors for your various parameters.

The likelihoods function sets the CMB qb values to 1 and all others (esp., residuals) to their maximum likelihood values. To instead include these parameters in your fit, you set the `prior` arguments. So:

* `r_prior`: set to [0, 'inf'] to inclue $r$ in the fit (or ['-inf', 'inf'] to impose no physical prior)
* `alpha_prior`: same as `r_prior`-- set to None to not fit for alpha template scalings.
* `res_prior`: same as previous-- will vary all $n_b$ parameters as part of the model
* `beam_prior`: this is different! To account for beam uncertainty, set this to [0, 1], where the first parameter is the Gaussian mean, and the second is the number of standard deviations to use as the width. Here, you must have set the `beam_error_product` in your config file to a dictionary containing the one-sigma error on the beam shape per ell, and then that envelope is varied according to a Gaussian with each step to marginalize over the beam uncertainty.

That wraps up the example!


Let's start here! If you can directly link to an image relevant to your notebook, such as [canonical logos](https://github.com/numpy/numpy/blob/main/doc/source/_static/numpylogo.svg), do so here at the top of your notebook. You can do this with Markdown syntax,

> ``

or edit this cell to see raw HTML `img` demonstration. This is preferred if you need to shrink your embedded image. **Either way be sure to include `alt` text for any embedded images to make your content more accessible.**

# Project Pythia Notebook Template

Next, title your notebook appropriately with a top-level Markdown header, `#`. Do not use this level header anywhere else in the notebook. Our book build process will use this title in the navbar, table of contents, etc. Keep it short, keep it descriptive. Follow this with a `---` cell to visually distinguish the transition to the prerequisites section.

---

## Overview
If you have an introductory paragraph, lead with it here! Keep it short and tied to your material, then be sure to continue into the required list of topics below,

1. This is a numbered list of the specific topics
1. These should map approximately to your main sections of content
1. Or each second-level, `##`, header in your notebook
1. Keep the size and scope of your notebook in check
1. And be sure to let the reader know up front the important concepts they'll be leaving with

## Prerequisites
This section was inspired by [this template](https://github.com/alan-turing-institute/the-turing-way/blob/master/book/templates/chapter-template/chapter-landing-page.md) of the wonderful [The Turing Way](https://the-turing-way.netlify.app/welcome.html) Jupyter Book.

Following your overview, tell your reader what concepts, packages, or other background information they'll **need** before learning your material. Tie this explicitly with links to other pages here in Foundations or to relevant external resources. Remove this body text, then populate the Markdown table, denoted in this cell with `|` vertical brackets, below, and fill out the information following. In this table, lay out prerequisite concepts by explicitly linking to other Foundations material or external resources, or describe generally helpful concepts.

Label the importance of each concept explicitly as **helpful/necessary**.

| Concepts | Importance | Notes |
| --- | --- | --- |
| [Intro to Cartopy](../core/cartopy/cartopy) | Necessary | |
| [Understanding of NetCDF](some-link-to-external-resource) | Helpful | Familiarity with metadata structure |
| Project management | Helpful | |

- **Time to learn**: estimate in minutes. For a rough idea, use 5 mins per subsection, 10 if longer; add these up for a total. Safer to round up and overestimate.
- **System requirements**:
    - Populate with any system, version, or non-Python software requirements if necessary
    - Otherwise use the concepts table above and the Imports section below to describe required packages as necessary
    - If no extra requirements, remove the **System requirements** point altogether

---

## Imports
Begin your body of content with another `---` divider before continuing into this section, then remove this body text and populate the following code cell with all necessary Python imports **up-front**:

```python
import sys
```

## Your first content section

This is where you begin your first section of material, loosely tied to your objectives stated up front. Tie together your notebook as a narrative, with interspersed Markdown text, images, and more as necessary,

```python
# as well as any and all of your code cells
print("Hello world!")
```

### A content subsection
Divide and conquer your objectives with Markdown subsections, which will populate the helpful navbar in Jupyter Lab and here on the Jupyter Book!

```python
# some subsection code
new = "helpful information"
```

### Another content subsection
Keep up the good work! A note, *try to avoid using code comments as narrative*, and instead let them only exist as brief clarifications where necessary.

## Your second content section
Here we can move on to our second objective, and we can demonstrate

### Subsection to the second section

#### a quick demonstration

##### of further and further

###### header levels

as well $m = a * t / h$ text! Similarly, you have access to other $\LaTeX$ equation [**functionality**](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Typesetting%20Equations.html) via MathJax (demo below from link),

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

Check out [**any number of helpful Markdown resources**](https://www.markdownguide.org/basic-syntax/) for further customizing your notebooks and the [**Jupyter docs**](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Working%20With%20Markdown%20Cells.html) for Jupyter-specific formatting information. Don't hesitate to ask questions if you have problems getting it to look *just right*.

## Last Section

If you're comfortable, and as we briefly used for our embedded logo up top, you can embed raw html into Jupyter Markdown cells (edit to see):

Info

    Your relevant information here!

Feel free to copy this around and edit or play around with yourself. Some other `admonitions` you can put in:

Success

    We got this done after all!

Warning

    Be careful!

Danger

    Scary stuff be here.

We also suggest checking out Jupyter Book's [brief demonstration](https://jupyterbook.org/content/metadata.html#jupyter-cell-tags) on adding cell tags to your cells in Jupyter Notebook, Lab, or manually. Using these cell tags can allow you to [customize](https://jupyterbook.org/interactive/hiding.html) how your code content is displayed and even [demonstrate errors](https://jupyterbook.org/content/execute.html#dealing-with-code-that-raises-errors) without altogether crashing our loyal army of machines!

---

## Summary
Add one final `---` marking the end of your body of content, and then conclude with a brief single paragraph summarizing at a high level the key pieces that were learned and how they tied to your objectives. Look to reiterate what the most important takeaways were.

### What's next?
Let Jupyter book tie this to the next (sequential) piece of content that people could move on to down below and in the sidebar. However, if this page uniquely enables your reader to tackle other nonsequential concepts throughout this book, or even external content, link to it here!

## Resources and references
Finally, be rigorous in your citations and references as necessary. Give credit where credit is due. Also, feel free to link to relevant external material, further reading, documentation, etc. Then you're done! Give yourself a quick review, a high five, and send us a Pull Request. A few final notes:
 - `Kernel > Restart Kernel and Run All Cells...` to confirm that your notebook will cleanly run from start to finish
 - `Kernel > Restart Kernel and Clear All Outputs...` before committing your notebook, our machines will do the heavy lifting
 - Take credit! Provide author contact information if you'd like; if so, consider adding information here at the bottom of your notebook
 - Give credit! Attribute appropriate authorship for referenced code, information, images, etc.
 - Only include what you're legally allowed: **no copyright infringement or plagiarism**

Thank you for your contribution!